Corpus ID: 211732635

MotherNets: Rapid Deep Ensemble Learning

@inproceedings{Wasay2020MotherNetsRD,
  title={MotherNets: Rapid Deep Ensemble Learning},
  author={A. Wasay and Brian Hentschel and Yuze Liao and Sanyuan Chen and Stratos Idreos},
  booktitle={MLSys},
  year={2020}
}
Ensembles of deep neural networks significantly improve generalization accuracy. However, training neural network ensembles requires a large amount of computational resources and time. State-of-the-art approaches either train all networks from scratch leading to prohibitive training cost that allows only very small ensemble sizes in practice, or generate ensembles by training a monolithic architecture, which results in lower model diversity and decreased prediction accuracy. We propose… Expand
CONVOLUTIONAL NEURAL NETWORK ENSEMBLES
Convolutional neural networks are utilized to solve increasingly more complex problems and with more data. As a result, researchers and practitioners seek to scale the representational power of suchExpand
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
TLDR
This study reviews recent advances in UQ methods used in deep learning and investigates the application of these methods in reinforcement learning (RL), and outlines a few important applications of UZ methods. Expand
CODE: COMPILER-BASED NEURON-AWARE ENSEMBLE TRAINING
Deep Neural Networks (DNNs) are redefining the state-of-the-art performance in a variety of tasks like speech recognition and image classification. These impressive results are often enabled byExpand
Classification of colorectal tissue images from high throughput tissue microarrays by ensemble deep learning methods
TLDR
The authors' high-accuracy algorithm for colorectal tissue classification in high-throughput TMAs is amenable to images from different institutions, core sizes and stain intensity, which helps to reduce error in TMA core evaluations with previously given labels. Expand
Deep Learning: Systems and Responsibility
TLDR
This tutorial on deep learning is presented, highlighting the data systems nature of neural networks as well as research opportunities for advancements through data management techniques, and essential considerations for responsible application of neural Networks in critical human-facing problems in society. Expand
Horizontally Fused Training Array: An Effective Hardware Utilization Squeezer for Training Novel Deep Learning Models
TLDR
HFTA is a new DL framework extension library that horizontally fuses the models from different repetitive jobs deeply down to operators and then trains them simultaneously on a shared accelerator and results indicate that HFTA is highly effective in improving hardware utilization and achieves up to 15.1× higher training throughput. Expand
MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworks
TLDR
It is shown that binary mixing in features particularly with rectangular patches from CutMix enhances results by making subnetworks stronger and more diverse, which improves state of the art for image classification on CIFAR-100 and Tiny ImageNet datasets. Expand
The Case for High-Accuracy Classification: Think Small, Think Many!
TLDR
This paper proposes an efficient and lightweight deep classification ensemble structure based on a combination of simple color features, particularly designed for “highaccuracy” image classifications with low false positives, for explosion detection use-case applied to images and videos. Expand
Understanding and optimizing packed neural network training for hyper-parameter tuning
TLDR
This paper proposes a primitive for jointly training multiple neural network models on a single GPU, called pack, and presents a comprehensive empirical study of pack and end-to-end experiments that suggest significant improvements for hyperparameter tuning. Expand
A strategy for developing a pair of diverse deep learning classifiers for use in a 2-classifier system
TLDR
A methodology of generating the label vector into two different forms by exploiting the similarity between the classes, which yields better performance in terms of both diversity and accuracy as compared to the standard approach of penalizing all the incorrect classes equally via binary label vector. Expand
...
1
2
...

References

SHOWING 1-10 OF 91 REFERENCES
The relative performance of ensemble methods with deep convolutional neural networks for image classification
TLDR
This work investigated multiple widely used ensemble methods, including unweighted averaging, majority voting, the Bayes Optimal Classifier, and the (discrete) Super Learner, for image recognition tasks, with deep neural networks as candidate algorithms. Expand
Densely Connected Convolutional Networks
TLDR
The Dense Convolutional Network (DenseNet), which connects each layer to every other layer in a feed-forward fashion, and has several compelling advantages: they alleviate the vanishing-gradient problem, strengthen feature propagation, encourage feature reuse, and substantially reduce the number of parameters. Expand
Snapshot Ensembles: Train 1, get M for free
TLDR
This paper proposes a method to obtain the seemingly contradictory goal of ensembling multiple neural networks at no additional training cost by training a single neural network, converging to several local minima along its optimization path and saving the model parameters. Expand
Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks
TLDR
It is demonstrated that TreeNets can improve ensemble performance and that diverse ensembles can be trained end-to-end under a unified loss, achieving significantly higher "oracle" accuracies than classical ensembled. Expand
Aws pricing
  • 2019
Accelerating learning via knowledge transfer
  • International Conference on Learning Representations (ICLR)
  • 2016
Deep Residual Learning for Image Recognition
TLDR
This work presents a residual learning framework to ease the training of networks that are substantially deeper than those used previously, and provides comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. Expand
Net2Net: Accelerating Learning via Knowledge Transfer
TLDR
The Net2Net technique accelerates the experimentation process by instantaneously transferring the knowledge from a previous network to each new deeper or wider network, and demonstrates a new state of the art accuracy rating on the ImageNet dataset. Expand
Network Morphism
TLDR
A systematic study on how to morph a well-trained neural network to a new one so that its network function can be completely preserved, and proposes a family of parametric-activation functions to facilitate the morphing of any continuous nonlinear activation neurons. Expand
Wide Residual Networks
TLDR
This paper conducts a detailed experimental study on the architecture of ResNet blocks and proposes a novel architecture where the depth and width of residual networks are decreased and the resulting network structures are called wide residual networks (WRNs), which are far superior over their commonly used thin and very deep counterparts. Expand
...
1
2
3
4
5
...