Corpus ID: 6500

Adaptive Neural Networks for Fast Test-Time Prediction

@article{Bolukbasi2017AdaptiveNN,
  title={Adaptive Neural Networks for Fast Test-Time Prediction},
  author={Tolga Bolukbasi and Joseph Wang and Ofer Dekel and Venkatesh Saligrama},
  journal={ArXiv},
  year={2017},
  volume={abs/1702.07811}
}
We present an approach to adaptively utilize deep neural networks in order to reduce the evaluation time on new examples without loss of classification performance. Rather than attempting to redesign or approximate existing networks, we propose two schemes that adaptively utilize networks. First, we pose an adaptive network evaluation scheme, where we learn a system to adaptively choose the components of a deep network to be evaluated for each example. By allowing examples correctly classified… Expand
Improved Techniques for Training Adaptive Deep Networks
TLDR
This paper considers a typical adaptive deep network with multiple intermediate classifiers and presents three techniques to improve its training efficacy from two aspects: a Gradient Equilibrium algorithm to resolve the conflict of learning of different classifiers; an Inline Subnetwork Collaboration approach and a One-for-all Knowledge Distillation algorithm to enhance the collaboration among classifiers. Expand
Learning Time/Memory-Efficient Deep Architectures with Budgeted Super Networks
TLDR
A novel family of models called Budgeted Super Networks (BSN) is proposed, learned using gradient descent techniques applied on a budgeted learning objective function which integrates a maximum authorized cost, while making no assumption on the nature of this cost. Expand
Multi-Scale Dense Convolutional Networks for Efficient Prediction
TLDR
A new convolutional neural network architecture with the ability to adapt dynamically to computational resource limits at test time and substantially improves the state-of-the-art in both settings is introduced. Expand
Multi-Scale Dense Networks for Resource Efficient Image Classification
TLDR
Experiments demonstrate that the proposed framework substantially improves the existing state-of-the-art in both image classification with computational resource limits at test time and budgeted batch classification. Expand
Learning Anytime Predictions in Neural Networks via Adaptive Loss Balancing
TLDR
This work optimize auxiliary losses jointly in an adaptive weighted sum, where the weights are inversely proportional to average of each loss, to achieve near-optimal anytime results at any budget, at the cost of a const fraction of extra computation. Expand
Runtime Neural Pruning
TLDR
A Runtime Neural Pruning (RNP) framework which prunes the deep neural network dynamically at the runtime and preserves the full ability of the original network and conducts pruning according to the input image and current feature maps adaptively. Expand
Anytime Recognition with Routing Convolutional Networks
TLDR
This work proposes a new Routing Convolutional Network (RCN), which adaptively selects the optimal layer as exit for a specific testing sample within a specific time budget and can be adapted to dense prediction tasks, e.g., scene parsing, to achieve the pixel-level anytime prediction. Expand
Sacrificing Accuracy for Reduced Computation: Cascaded Inference Based on Softmax Confidence
TLDR
Basing of early termination on softmax classifier outputs is justified by experimentation that demonstrates an almost linear relation between confidence levels in intermediate classifiers and accuracy. Expand
Overview of Configuring Adaptive Activation Functions for Deep Neural Networks - A Comparative Study
Recently, the deep neural networks (DNN) have demonstrated many performances in the pattern recognition paradigm. The research studies on DNN include depth layer networks, filters, training andExpand
SGAD: Soft-Guided Adaptively-Dropped Neural Network
TLDR
An innovative soft-guided adaptively-dropped (SGAD) neural network is proposed in this paper that can reduce the FLOPs by 77% with less than 1% drop in accuracy on CIFAR-10. Expand
...
1
2
3
...

References

SHOWING 1-10 OF 27 REFERENCES
Distilling the Knowledge in a Neural Network
TLDR
This work shows that it can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model and introduces a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse. Expand
Deep Residual Learning for Image Recognition
TLDR
This work presents a residual learning framework to ease the training of networks that are substantially deeper than those used previously, and provides comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. Expand
Learning Structured Sparsity in Deep Neural Networks
TLDR
The results show that for CIFAR-10, regularization on layer depth can reduce 20 layers of a Deep Residual Network to 18 layers while improve the accuracy from 91.25% to 92.60%, which is still slightly higher than that of original ResNet with 32 layers. Expand
Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
TLDR
A binary matrix multiplication GPU kernel is programmed with which it is possible to run the MNIST QNN 7 times faster than with an unoptimized GPU kernel, without suffering any loss in classification accuracy. Expand
Efficient Learning by Directed Acyclic Graph For Resource Constrained Prediction
TLDR
This work models the problem of reducing test-time acquisition costs in classification systems as a directed acyclic graph (DAG) where internal nodes correspond to sensor subsets and decision functions at each node choose whether to acquire a new sensor or classify using the available measurements. Expand
Sparse Convolutional Neural Networks
TLDR
This work shows how to reduce the redundancy in these parameters using a sparse decomposition, and proposes an efficient sparse matrix multiplication algorithm on CPU for Sparse Convolutional Neural Networks (SCNN) models. Expand
LCNN: Lookup-Based Convolutional Neural Network
TLDR
This paper introduces LCNN, a lookup-based convolutional neural network that encodes convolutions by few lookups to a dictionary that is trained to cover the space of weights in CNNs and shows the benefits of LCNN in few-shot learning and few-iteration learning, two crucial aspects of on-device training of deep learning models. Expand
ImageNet classification with deep convolutional neural networks
TLDR
A large, deep convolutional neural network was trained to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes and employed a recently developed regularization method called "dropout" that proved to be very effective. Expand
Binarized Neural Networks
TLDR
A binary matrix multiplication GPU kernel is written with which it is possible to run the MNIST BNN 7 times faster than with an unoptimized GPU kernel, without suffering any loss in classification accuracy. Expand
BinaryConnect: Training Deep Neural Networks with binary weights during propagations
TLDR
BinaryConnect is introduced, a method which consists in training a DNN with binary weights during the forward and backward propagations, while retaining precision of the stored weights in which gradients are accumulated, and near state-of-the-art results with BinaryConnect are obtained on the permutation-invariant MNIST, CIFAR-10 and SVHN. Expand
...
1
2
3
...