Corpus ID: 5501937

The Incredible Shrinking Neural Network: New Perspectives on Learning Representations Through The Lens of Pruning

@article{Wolfe2017TheIS,
  title={The Incredible Shrinking Neural Network: New Perspectives on Learning Representations Through The Lens of Pruning},
  author={Nikolas Wolfe and Aditya Sharma and Lukas Drude and B. Raj},
  journal={ArXiv},
  year={2017},
  volume={abs/1701.04465}
}
How much can pruning algorithms teach us about the fundamentals of learning representations in neural networks? A lot, it turns out. Neural network model compression has become a topic of great interest in recent years, and many different techniques have been proposed to address this problem. In general, this is motivated by the idea that smaller models typically lead to better generalization. At the same time, the decision of what to prune and when to prune necessarily forces us to confront… Expand
Learning Neural Network Classifiers with Low Model Complexity
TLDR
A continuous and differentiable error functional for a neural network that minimizes its empirical error as well as a measure of the model complexity, which strongly suggest that deep learning techniques can benefit from model complexity control methods such as the LCNN learning rule. Expand
FeTa: A DCA Pruning Algorithm with Generalization Error Guarantees
TLDR
This work proposes a cheap pruning algorithm for fully connected DNN layers based on difference of convex functions (DC) optimisation, that requires little or no retraining, and provides a theoretical analysis for the growth in the Generalization Error of a DNN for the case of bounded perturbations to the hidden layers. Expand
Lifelong Learning Starting From Zero
TLDR
This work presents a deep neural-network model for lifelong learning inspired by several forms of neuroplasticity, and analyzes it from the perspective of accuracy, energy efficiency, and versatility and compares it to other network models, finding better performance in several cases. Expand
Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon
TLDR
It is proved that the final prediction performance drop after pruning is bounded by a linear combination of the reconstructed errors caused at each layer, so there is a guarantee that one only needs to perform a light retraining process on the pruned network to resume its original prediction performance. Expand
Smaller Models, Better Generalization
TLDR
This paper proposes a novel loss function that helps in achieving considerable sparsity at comparable accuracies to that of dense models and shows the superiority of the method in achieving sparser models that generalize well. Expand
Auto-Balanced Filter Pruning for Efficient Convolutional Neural Networks
TLDR
This paper proposes an iterative approach named Auto-balanced Filter Pruning, where the network is pre-trained in an innovative auto-balanced way to transfer the representational capacity of its convolutional layers to a fraction of the filters, prune the redundant ones, then re-train it to restore the accuracy. Expand
Channel-wise pruning of neural networks with tapering resource constraint
TLDR
This work proposes a new method for compute-constrained structured channel-wise pruning of convolutional neural networks, which iteratively fine-tunes the network, while gradually tapering the computation resources available to the pruned network via a holonomic constraint in the method of Lagrangian multipliers framework. Expand
Leveraging Filter Correlations for Deep Model Compression
TLDR
The proposed compression method yields state-of-the-art FLOPs compression rates on various benchmarks, such as LeNet-5, VGG-16, and ResNet-50,56, while still achieving excellent predictive performance for tasks such as object detection on benchmark datasets. Expand
Approximated Oracle Filter Pruning for Destructive CNN Width Optimization
TLDR
Approximated Oracle Filter Pruning (AOFP) is proposed, which keeps searching for the least important filters in a binary search manner, makes pruning attempts by masking out filters randomly, accumulates the resulting errors, and finetunes the model via a multi-path framework. Expand
SubFlow: A Dynamic Induced-Subgraph Strategy Toward Real-Time DNN Inference and Training
  • Seulki Lee, S. Nirjon
  • Computer Science
  • 2020 IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS)
  • 2020
TLDR
SubFlow-a dynamic adaptation and execution strategy for a deep neural network (DNN), which enables real-time DNN inference and training is introduced and open-source SubFlow is implemented by extending TensorFlow with full compatibility by adding SubFlow operations for convolutional and fully-connected layers of a DNN. Expand
...
1
2
...

References

SHOWING 1-10 OF 21 REFERENCES
Dropout: a simple way to prevent neural networks from overfitting
TLDR
It is shown that dropout improves the performance of neural networks on supervised learning tasks in vision, speech recognition, document classification and computational biology, obtaining state-of-the-art results on many benchmark data sets. Expand
Fault tolerance of pruned multilayer networks
  • B. Segee, M.J. Carter
  • Computer Science
  • IJCNN-91-Seattle International Joint Conference on Neural Networks
  • 1991
TLDR
Assessment of the tolerance of multilayer feedforward networks to the zeroing of individual weights finds the unpruned networks, which had considerably more free parameters, were found to be no more tolerant to weight zeroing than the pruned networks. Expand
Second Order Derivatives for Network Pruning: Optimal Brain Surgeon
TLDR
Of OBS, Optimal Brain Damage, and magnitude-based methods, only OBS deletes the correct weights from a trained XOR network in every case, and thus yields better generalization on test data. Expand
What Size Net Gives Valid Generalization?
TLDR
It is shown that if m O(W/ ∊ log N/∊) random examples can be loaded on a feedforward network of linear threshold functions with N nodes and W weights, so that at least a fraction 1 ∊/2 of the examples are correctly classified, then one has confidence approaching certainty that the network will correctly classify a fraction 2 ∊ of future test examples drawn from the same distribution. Expand
Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding
TLDR
This work introduces "deep compression", a three stage pipeline: pruning, trained quantization and Huffman coding, that work together to reduce the storage requirement of neural networks by 35x to 49x without affecting their accuracy. Expand
Maxout Networks
TLDR
A simple new model called maxout is defined designed to both facilitate optimization by dropout and improve the accuracy of dropout's fast approximate model averaging technique. Expand
Pruning algorithms-a survey
  • Russell Reed
  • Computer Science, Medicine
  • IEEE Trans. Neural Networks
  • 1993
TLDR
The approach taken by the methods described here is to train a network that is larger than necessary and then remove the parts that are not needed. Expand
Learning with limited numerical precision using the cascade-correlation algorithm
TLDR
An empirical study of the effects of limited precision in cascade-correlation networks on three different learning problems is presented and techniques for dynamic rescaling and probabilistic rounding that allow reliable convergence down to 7 bits of precision or less are introduced. Expand
Skeletonization: A Technique for Trimming the Fat from a Network via Relevance Assessment
TLDR
The basic idea is to iteratively train the network to a certain performance criterion, compute a measure of relevance that identifies which input or hidden units are most critical to performance, and automatically trim the least relevant units. Expand
The Cascade-Correlation Learning Architecture
TLDR
The Cascade-Correlation architecture has several advantages over existing algorithms: it learns very quickly, the network determines its own size and topology, it retains the structures it has built even if the training set changes, and it requires no back-propagation of error signals through the connections of the network. Expand
...
1
2
3
...