Corpus ID: 167217877

CGaP: Continuous Growth and Pruning for Efficient Deep Learning

@article{Du2019CGaPCG,
  title={CGaP: Continuous Growth and Pruning for Efficient Deep Learning},
  author={Xiaocong Du and Zheng Li and Yu Cao},
  journal={ArXiv},
  year={2019},
  volume={abs/1905.11533}
}
Today a canonical approach to reduce the computation cost of Deep Neural Networks (DNNs) is to pre-define an over-parameterized model before training to guarantee the learning capacity, and then prune unimportant learning units (filters and neurons) during training to improve model compactness. We argue it is unnecessary to introduce redundancy at the beginning of the training but then reduce redundancy for the ultimate inference model. In this paper, we propose a Continuous Growth and Pruning… Expand
Alternate Model Growth and Pruning for Efficient Training of Recommendation Systems
TLDR
This work proposes a dynamic training scheme, namely alternate model growth and pruning, to alternatively construct and prune weights in the course of training to reduce computation cost without hurting the model capacity at the end of offline training. Expand
Stacked Broad Learning System: From Incremental Flatted Structure to Deep Model
TLDR
Results imply that the proposed structure could highly reduce the number of nodes and the training time of the original BLS in the classification task of some datasets, and outperforms the selected state-of-the-art methods on both accuracy and training speed. Expand
AutoGrow: Automatic Layer Growing in Deep Convolutional Networks
TLDR
This work proposesAutoGrow to automate depth discovery in DNNs: starting from a shallow seed architecture, AutoGrow grows new layers if the growth improves the accuracy; otherwise, stops growing and thus discovers the depth. Expand
PID Controller-Based Stochastic Optimization Acceleration for Deep Neural Networks
TLDR
The proposed proportional-integral-derivative (PID) approach alleviates the overshoot problem suffered by stochastic gradient descent-momentum, and demonstrates the effectiveness of the method on benchmark data sets, including CIFAR10, CifAR100, Tiny-ImageNet, and PTB. Expand
MDLdroidLite: a release-and-inhibit control approach to resource-efficient deep neural networks on mobile devices
TLDR
This paper proposes a novel Release-and-Inhibit Control (RIC) approach based on Model Predictive Control theory to efficiently grow DNNs from tiny to backbone, and designs a gate-based fast adaptation mechanism for channel-level knowledge transformation to quickly adapt new-born neurons with existing neurons, enabling safe parameter adaptation and fast convergence for on-device training. Expand
INVITED: Computation on Sparse Neural Networks and its Implications for Future Hardware
TLDR
This paper summarizes the current status of the research on the computation of sparse neural networks, from the perspective of the sparse algorithms, the software frameworks, and the hardware accelerations, and observes that the search for the sparse structure can be a general methodology for high-quality model explorations, in addition to a strategy forhigh-efficiency model execution. Expand
Computation on Sparse Neural Networks: an Inspiration for Future Hardware
TLDR
This paper summarizes the current status of the research on the computation of sparse neural networks, from the perspective of the sparse algorithms, the software frameworks, and the hardware accelerations, and observes that the search for the sparse structure can be a general methodology for high-quality model explorations, in addition to a strategy forhigh-efficiency model execution. Expand

References

SHOWING 1-10 OF 42 REFERENCES
Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
TLDR
This paper introduces network trimming which iteratively optimizes the network by pruning unimportant neurons based on analysis of their outputs on a large dataset, inspired by an observation that the outputs of a significant portion of neurons in a large network are mostly zero. Expand
Rethinking the Value of Network Pruning
TLDR
It is found that with optimal learning rate, the "winning ticket" initialization as used in Frankle & Carbin (2019) does not bring improvement over random initialization, and the need for more careful baseline evaluations in future research on structured pruning methods is suggested. Expand
Pruning Convolutional Neural Networks for Resource Efficient Inference
TLDR
It is shown that pruning can lead to more than 10x theoretical (5x practical) reduction in adapted 3D-convolutional filters with a small drop in accuracy in a recurrent gesture classifier. Expand
NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm
TLDR
A network growth algorithm that complements network pruning to learn both weights and compact DNN architectures during training, and delivers significant additional parameter and FLOPs reduction relative to pruning-only methods. Expand
Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks
TLDR
The proposed Soft Filter Pruning (SFP) method enables the pruned filters to be updated when training the model after pruning, which has two advantages over previous works: larger model capacity and less dependence on the pretrained model. Expand
Learning both Weights and Connections for Efficient Neural Network
TLDR
A method to reduce the storage and computation required by neural networks by an order of magnitude without affecting their accuracy by learning only the important connections, and prunes redundant connections using a three-step method. Expand
ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression
TLDR
ThiNet is proposed, an efficient and unified framework to simultaneously accelerate and compress CNN models in both training and inference stages, and it is revealed that it needs to prune filters based on statistics information computed from its next layer, not the current layer, which differentiates ThiNet from existing methods. Expand
Learning Structured Sparsity in Deep Neural Networks
TLDR
The results show that for CIFAR-10, regularization on layer depth can reduce 20 layers of a Deep Residual Network to 18 layers while improve the accuracy from 91.25% to 92.60%, which is still slightly higher than that of original ResNet with 32 layers. Expand
Dynamic Network Surgery for Efficient DNNs
TLDR
A novel network compression method called dynamic network surgery, which can remarkably reduce the network complexity by making on-the-fly connection pruning by proving that it outperforms the recent pruning method by considerable margins. Expand
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
TLDR
Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin. Expand
...
1
2
3
4
5
...