Recovering from Random Pruning: On the Plasticity of Deep Convolutional Neural Networks

@article{Mittal2018RecoveringFR,
  title={Recovering from Random Pruning: On the Plasticity of Deep Convolutional Neural Networks},
  author={Deepak Mittal and Shweta Bhardwaj and Mitesh M. Khapra and Balaraman Ravindran},
  journal={2018 IEEE Winter Conference on Applications of Computer Vision (WACV)},
  year={2018},
  pages={848-857}
}
Recently there has been a lot of work on pruning filters from deep convolutional neural networks (CNNs) with the intention of reducing computations. The key idea is to rank the filters based on a certain criterion (say, l1-norm, average percentage of zeros, etc) and retain only the top ranked filters. Once the low scoring filters are pruned away the remainder of the network is fine tuned and is shown to give performance comparable to the original unpruned network. In this work, we report… 

Figures and Tables from this paper

An Efficient Pipeline for Pruning Convolutional Neural Networks
  • Chengcheng Li, Zi Wang, H. Qi
  • Computer Science
    2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA)
  • 2020
TLDR
This paper presents a computationally efficient framework for structured pruning by exploring the potential of leveraging the intermediate results generated during the fine-tuning step to rank the importance of filters and thus converting a three-step pipeline to a two- step pipeline, resulting in significant savings in computation time.
Pruning Filters while Training for Efficiently Optimizing Deep Learning Networks
TLDR
This work proposes a dynamic pruning-while-training procedure, wherein filters of the convolutional layers of a deep neural network are pruned during training itself, thereby precluding the need for separate retraining.
Evaluating the Merits of Ranking in Structured Network Pruning
TLDR
This work further explores the value of the ranking criteria in pruning to show that if channels are removed gradually and iteratively, alternating with fine-tuning on the target dataset, ranking criteria are indeed not necessary to select redundant channels, and proposes a GFLOPs-aware iterative pruning strategy that can further lead to lower inference time by 15% without sacrificing accuracy.
Intriguing Properties of Randomly Weighted Networks: Generalizing While Learning Next to Nothing
TLDR
This paper proposes to fix almost all layers of a deep convolutional neural network, allowing only a small portion of the weights to be learned, and suggests practical ways to harness it to create more robust and compact representations.
Out-of-the-box channel pruned networks
TLDR
A Reinforcement Learning (RL) policy-based search algorithm with a direct objective of finding transferable layer-wise pruning profiles using many models for the same architecture is developed and results show that the transferred RL-based profiles are as good or better than best profiles found on the original dataset via exhaustive search.
Convolutional Neural Network Pruning with Structural Redundancy Reduction
TLDR
This study first statistically model the network pruning problem in a redundancy reduction perspective and finds that pruning in the layer(s) with the most structural redundancy outperforms pruning the least important filters across all layers.
Neural Pruning via Growing Regularization
TLDR
This work proposes an L2 regularization variant with rising penalty factors and shows it can bring significant accuracy gains compared with its one-shot counterpart, even when the same weights are removed.
Discriminative Layer Pruning for Convolutional Neural Networks
TLDR
It is shown that a simple subspace projection approach can be employed to estimate the importance of network layers, enabling the pruning of CNNs to a resource-efficient depth within a given network size constraint, and that cascading discriminative layer pruning with filter-oriented pruning improves the resource-efficiency of the resulting network.
...
...

References

SHOWING 1-10 OF 52 REFERENCES
Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
TLDR
This paper introduces network trimming which iteratively optimizes the network by pruning unimportant neurons based on analysis of their outputs on a large dataset, inspired by an observation that the outputs of a significant portion of neurons in a large network are mostly zero.
Pruning Filters for Efficient ConvNets
TLDR
This work presents an acceleration method for CNNs, where it is shown that even simple filter pruning techniques can reduce inference costs for VGG-16 and ResNet-110 by up to 38% on CIFAR10 while regaining close to the original accuracy by retraining the networks.
Pruning Convolutional Neural Networks for Resource Efficient Inference
TLDR
It is shown that pruning can lead to more than 10x theoretical (5x practical) reduction in adapted 3D-convolutional filters with a small drop in accuracy in a recurrent gesture classifier.
ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression
TLDR
ThiNet is proposed, an efficient and unified framework to simultaneously accelerate and compress CNN models in both training and inference stages, and it is revealed that it needs to prune filters based on statistics information computed from its next layer, not the current layer, which differentiates ThiNet from existing methods.
Convolutional neural networks with low-rank regularization
TLDR
A new algorithm for computing the low-rank tensor decomposition for removing the redundancy in the convolution kernels and is more effective than iterative methods for speeding up large CNNs.
Channel Pruning for Accelerating Very Deep Neural Networks
  • Yihui He, X. Zhang, Jian Sun
  • Computer Science
    2017 IEEE International Conference on Computer Vision (ICCV)
  • 2017
TLDR
This paper proposes an iterative two-step algorithm to effectively prune each layer, by a LASSO regression based channel selection and least square reconstruction, and generalizes this algorithm to multi-layer and multi-branch cases.
Learning Structured Sparsity in Deep Neural Networks
TLDR
The results show that for CIFAR-10, regularization on layer depth can reduce 20 layers of a Deep Residual Network to 18 layers while improve the accuracy from 91.25% to 92.60%, which is still slightly higher than that of original ResNet with 32 layers.
An Entropy-based Pruning Method for CNN Compression
TLDR
This paper aims to simultaneously accelerate and compress off-the-shelf CNN models via filter pruning strategy with the proposed entropy-based method, which can reduce the size of intermediate activations.
Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding
TLDR
This work introduces "deep compression", a three stage pipeline: pruning, trained quantization and Huffman coding, that work together to reduce the storage requirement of neural networks by 35x to 49x without affecting their accuracy.
Training CNNs with Low-Rank Filters for Efficient Image Classification
TLDR
A new method for creating computationally efficient convolutional neural networks (CNNs) by using low-rank representations of Convolutional filters rather than approximating filters in previously-trained networks with more efficient versions, which shows similar or higher accuracy than conventional CNNs with much less compute.
...
...