Channel Pruning for Accelerating Very Deep Neural Networks

@article{He2017ChannelPF,
  title={Channel Pruning for Accelerating Very Deep Neural Networks},
  author={Yihui He and Xiangyu Zhang and Jian Sun},
  journal={2017 IEEE International Conference on Computer Vision (ICCV)},
  year={2017},
  pages={1398-1406}
}
  • Yihui He, X. Zhang, Jian Sun
  • Published 19 July 2017
  • Computer Science
  • 2017 IEEE International Conference on Computer Vision (ICCV)
In this paper, we introduce a new channel pruning method to accelerate very deep convolutional neural networks. [] Key Method We further generalize this algorithm to multi-layer and multi-branch cases. Our method reduces the accumulated error and enhance the compatibility with various architectures. Our pruned VGG-16 achieves the state-of-the-art results by 5× speed-up along with only 0.3% increase of error. More importantly, our method is able to accelerate modern networks like ResNet, Xception and suffers…

Figures and Tables from this paper

Linearly Replaceable Filters for Deep Network Channel Pruning
TLDR
A novel channel pruning method, Linearly Replaceable Filter (LRF), is proposed, which suggests that a filter that can be approximated by the linear combination of other filters is replaceable.
A Hybrid Statistics-based Channel Pruning Method for Deep CNNs*
  • Yan Zhou, Guangyi Liu, Dongli Wang
  • Computer Science
    2019 IEEE International Conference on Systems, Man and Cybernetics (SMC)
  • 2019
TLDR
A channel pruning method to simultaneously accelerate and compress deep CNNs while maintaining their accuracy and to achieve stronger generalization ability is proposed.
Compression of Deep Convolutional Neural Networks Using Effective Channel Pruning
TLDR
The proposed channel pruning criterion adopts the loss-approximating Taylor expansion based on not the pruned parameters but the parameters in the subsequent convolutional layer, which differentiates this method from existing methods, to evaluate the importance of each channel.
Variational Convolutional Neural Network Pruning
TLDR
Variational technique is introduced to estimate distribution of a newly proposed parameter, called channel saliency, based on which redundant channels can be removed from model via a simple criterion, and results in significant size reduction and computation saving.
Accelerating Deep Convnets via Sparse Subspace Clustering
TLDR
This paper proposes a novel filter pruning method that reduces the redundancy in one convolutional layer by applying sparse subspace clustering to its output feature maps and achieves state-of-the-art results after fine-tuning.
Exploiting Channel Similarity for Accelerating Deep Convolutional Neural Networks
TLDR
This work argues that channels revealing similar feature information have functional overlap and that most channels within each such similarity group can be removed without compromising model's representational power.
Dynamic Channel Pruning: Feature Boosting and Suppression
TLDR
This paper proposes feature boosting and suppression (FBS), a new method to predictively amplify salient convolutional channels and skip unimportant ones at run-time, and compares FBS to a range of existing channel pruning and dynamic execution schemes and demonstrates large improvements on ImageNet classification.
Channel Pruning for Accelerating Convolutional Neural Networks via Wasserstein Metric
TLDR
This paper proposes a novel channel pruning method via the Wasserstein metric that outperforms the existing methods on popular benchmarks and various network architectures.
A Weight-Based Channel Pruning Algorithm for Depth-Wise Separable Convolution Unit
TLDR
This paper adapts the channel pruning method to features of the depth-wise separable convolution unit and its variant and shows higher accuracy on the CIFAR-10 dataset than related work in some cases.
Structured Deep Neural Network Pruning by Varying Regularization Parameters
TLDR
A theoretically sound regularization-based pruning method to incrementally assign different regularization parameters to different weights based on their importance to the network, which can achieve 4x theoretical speedup with similar accuracies compared with the baselines.
...
...

References

SHOWING 1-10 OF 59 REFERENCES
Data-free Parameter Pruning for Deep Neural Networks
TLDR
It is shown how similar neurons are redundant, and a systematic way to remove them is proposed, which can be applied on top of most networks with a fully connected layer to give a smaller network.
Speeding up Convolutional Neural Networks with Low Rank Expansions
TLDR
Two simple schemes for drastically speeding up convolutional neural networks are presented, achieved by exploiting cross-channel or filter redundancy to construct a low rank basis of filters that are rank-1 in the spatial domain.
Going deeper with convolutions
We propose a deep convolutional neural network architecture codenamed Inception that achieves the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition
Accelerating Very Deep Convolutional Networks for Classification and Detection
TLDR
This paper aims to accelerate the test-time computation of convolutional neural networks, especially very deep CNNs, and develops an effective solution to the resulting nonlinear optimization problem without the need of stochastic gradient descent (SGD).
Less Is More: Towards Compact CNNs
TLDR
This work shows that, by incorporating sparse constraints into the objective function, it is possible to decimate the number of neurons during the training stage, thus theNumber of parameters and the memory footprint of the neural network are reduced, which is desirable at the test time.
Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
TLDR
A simple two-step approach for speeding up convolution layers within large convolutional neural networks based on tensor decomposition and discriminative fine-tuning is proposed, leading to higher obtained CPU speedups at the cost of lower accuracy drops for the smaller of the two networks.
Structured Pruning of Deep Convolutional Neural Networks
TLDR
The proposed work shows that when pruning granularities are applied in combination, the CIFAR-10 network can be pruned by more than 70% with less than a 1% loss in accuracy.
ImageNet classification with deep convolutional neural networks
TLDR
A large, deep convolutional neural network was trained to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes and employed a recently developed regularization method called "dropout" that proved to be very effective.
Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
TLDR
This paper introduces network trimming which iteratively optimizes the network by pruning unimportant neurons based on analysis of their outputs on a large dataset, inspired by an observation that the outputs of a significant portion of neurons in a large network are mostly zero.
Sparse Convolutional Neural Networks
TLDR
This work shows how to reduce the redundancy in these parameters using a sparse decomposition, and proposes an efficient sparse matrix multiplication algorithm on CPU for Sparse Convolutional Neural Networks (SCNN) models.
...
...