Leveraging Filter Correlations for Deep Model Compression

@article{Singh2020LeveragingFC,
  title={Leveraging Filter Correlations for Deep Model Compression},
  author={Pravendra Singh and Vinay Kumar Verma and Piyush Rai and Vinay P. Namboodiri},
  journal={2020 IEEE Winter Conference on Applications of Computer Vision (WACV)},
  year={2020},
  pages={824-833}
}
We present a filter correlation based model compression approach for deep convolutional neural networks. Our approach iteratively identifies pairs of filters with the largest pairwise correlations and drops one of the filters from each such pair. However, instead of discarding one of the filters from each such pair naïvely, the model is re-optimized to make the filters in these pairs maximally correlated, so that discarding one of the filters from the pair results in minimal information loss… 
Deep Model Compression based on the Training History
TLDR
A novel “History Based Filter Pruning (HBFP)” method that utilizes network training history for filter pruning and outperforms the state-of-the-art in terms of FLOPs reduction (floatingpoint operations) while maintaining the less error rate.
ACSL: Adaptive correlation-driven sparsity learning for deep neural network compression
TLDR
An adaptive correlation-driven sparsity learning (ACSL) framework for channel pruning that outperforms state-of-the-art methods on both image-level and pixel-level tasks and demonstrates the superiority of the proposed approach on three popular vision tasks.
Network Compression via Central Filter
TLDR
This paper proposes a novel filter pruning method—Central Filter (CF), which suggests that a filter is approximately equal to a set of other filters after appropriate adjustments, based on the discovery that the average similarity between feature maps changes very little, regardless of the number of input images.
Feature Variance Ratio-Guided Channel Pruning for Deep Convolutional Network Acceleration
TLDR
To recognize redundancies more accurately and therefore, accelerate networks better, a novel channel pruning criterion based on the Pearson correlation coefficient is proposed, which preserves the features that are essentially informative to the given task and avoids the influence of useless parameter scales.
Where to Prune: Using LSTM to Guide Data-Dependent Soft Pruning
TLDR
A new channel pruning framework is proposed, which can significantly reduce the computational complexity while maintaining sufficient model accuracy, and a data-dependent soft pruning method, dubbed Squeeze-Excitation-Pruning (SEP), which does not physically prune any filters but selectively excludes some kernels involved in calculating forward and backward propagations depending on the pruning scheme.
Meta Filter Pruning to Accelerate Deep Convolutional Neural Networks
TLDR
First, as a complement to the existing p-norm criterion, a new pruning criterion considering the filter relation via filter distance is introduced, and a meta pruning framework for filter pruning is built, so that the method could adaptively select the most appropriate pruned criterion as the filter distribution changes.
Compressing Deep CNNs using Basis Representation and Spectral Fine-tuning
We propose an efficient and straightforward method for compressing deep convolutional neural networks (CNNs) that uses basis filters to represent the convolutional layers, and optimizes the
A Generalized Zero-Shot Quantization of Deep Convolutional Neural Networks via Learned Weights Statistics
TLDR
This work is the first attempt towards the post-training zero- shot quantization of futuristic unnormalized deep neural networks and utilizes the distribution of the pretrained weights to assist the process of zero-shot quantization.
Channel Pruning for Accelerating Convolutional Neural Networks via Wasserstein Metric
TLDR
This paper proposes a novel channel pruning method via the Wasserstein metric that outperforms the existing methods on popular benchmarks and various network architectures.
Centripetal SGD for Pruning Very Deep Convolutional Networks With Complicated Structure
TLDR
Centripetal SGD (C-SGD), a novel optimization method, which can train several filters to collapse into a single point in the parameter hyperspace, is proposed, which partly solved an open problem of constrained filter pruning on CNNs with complicated structure, where some layers must be pruned following the others.
...
1
2
3
4
...

References

SHOWING 1-10 OF 72 REFERENCES
Stability Based Filter Pruning for Accelerating Deep CNNs
TLDR
This work presents a stability-based approach for filter-level pruning of CNNs that reduces the number of FLOPS and GPU memory footprint and is significantly outperforming other state-of-the-art filter pruning methods.
Multi-layer Pruning Framework for Compressing Single Shot MultiBox Detector
TLDR
A framework for compressing state-of-the-art Single Shot MultiBox Detector (SSD) in the following stages: Sparsity Induction, Filter Selection, and Filter Pruning, which results in the model with the size smaller than most existing compact architectures.
ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression
TLDR
ThiNet is proposed, an efficient and unified framework to simultaneously accelerate and compress CNN models in both training and inference stages, and it is revealed that it needs to prune filters based on statistics information computed from its next layer, not the current layer, which differentiates ThiNet from existing methods.
CLIP-Q: Deep Network Compression Learning by In-parallel Pruning-Quantization
TLDR
The proposed CLIP-Q method (Compression Learning by In-Parallel Pruning-Quantization) compresses AlexNet, GoogLeNet, and ResNet-50 by 10-fold, while preserving the uncompressed network accuracies on ImageNet, to take advantage of the complementary nature of pruning and quantization.
Play and Prune: Adaptive Filter Pruning for Deep Model Compression
TLDR
This work presents a new min-max framework for filter-level pruning of CNNs, which reduces the number of parameters of VGG-16 by an impressive factor of 17.5X, and number of FLOPS by 6.43X, with no loss of accuracy, significantly outperforming other state-of-the-art filter pruning methods.
Exploring Linear Relationship in Feature Map Subspace for ConvNets Compression
TLDR
This paper proposes a novel filter pruning method to compress and accelerate CNNs based on the linear relationship identified in different feature map subspaces via visualization of feature maps, which eliminates the redundancy in convolutional filters by applying subspace clustering to feature maps.
Coreset-Based Neural Network Compression
TLDR
A novel Convolutional Neural Network compression algorithm based on coreset representations of filters that provides AlexNet-like accuracy, with a memory footprint that is 832\(\times \) smaller than the original AlexNet, while also introducing significant reductions in inference time as well.
Model compression via distillation and quantization
TLDR
This paper proposes two new compression methods, which jointly leverage weight quantization and distillation of larger teacher networks into smaller student networks, and shows that quantized shallow students can reach similar accuracy levels to full-precision teacher models.
Auto-Balanced Filter Pruning for Efficient Convolutional Neural Networks
TLDR
This paper proposes an iterative approach named Auto-balanced Filter Pruning, where the network is pre-trained in an innovative auto-balanced way to transfer the representational capacity of its convolutional layers to a fraction of the filters, prune the redundant ones, then re-train it to restore the accuracy.
Pruning Filter via Geometric Median for Deep Convolutional Neural Networks Acceleration
TLDR
This paper proposes a novel filter pruning method, namely Filter Pruning via Geometric Median (FPGM), to compress the model regardless of those two requirements, which have advanced the state-of-the-art CNN models.
...
1
2
3
4
5
...