Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups

@article{Ioannou2016DeepRI,
  title={Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups},
  author={Yani Andrew Ioannou and Duncan P. Robertson and Roberto Cipolla and Antonio Criminisi},
  journal={2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2016},
  pages={5977-5986}
}
We propose a new method for creating computationally efficient and compact convolutional neural networks (CNNs) using a novel sparse connection structure that resembles a tree root. [] Key Result Our results show similar or higher accuracy than the baseline architectures with much less computation, as measured by CPU and GPU timings. For example, for ResNet 50, our model has 40% fewer parameters, 45% fewer floating point operations, and is 31% (12%) faster on a CPU (GPU).
View on IEEE
arxiv.org
191 Citations
Highly Influential Citations
17
Background Citations
107
Methods Citations
63
Results Citations
3

Figures and Tables from this paper

191 Citations

A Pre-defined Sparse Kernel Based Convolution for Deep CNNs

The high demand for computational and storage resources severely impede the deployment of deep convolutional neural networks (CNNs) in limited-resource devices. Recent CNN architectures have proposed…

pSConv: A Pre-defined S parse Kernel Based Convolution for Deep CNNs

The high demand for computational and storage resources severely impedes the deployment of deep convolutional neural networks (CNNs) in limited resource devices. Recent CNN architectures have…

SBNet: Sparse Blocks Network for Fast Inference

This work leverages the sparsity structure of computation masks and proposes a novel tiling-based sparse convolution algorithm that is effective on LiDAR-based 3D object detection, and reports significant wall-clock speed-ups compared to dense convolution without noticeable loss of accuracy.

Training CNNs with Selective Allocation of Channels

This paper modifications a standard convolutional layer to have a new functionality of channel-selectivity, so that the layer is trained to select important channels to re-distribute their parameters, without adding more parameters.

DualConv: Dual Convolutional Kernels for Lightweight Deep Neural Networks

Experimental results show that, combined with the structural innovations, DualConv significantly reduces the computational cost and number of parameters of deep neural networks while surprisingly achieving slightly higher accuracy than the original models in some cases.

ADNet: Adaptively Dense Convolutional Neural Networks

This paper presents a layer attention based Adaptively Dense Network (ADNet) by adaptively determining the reuse status of hierarchical preceding features by adaptingively determiningThe reuse status is adapted to fuse multi-level internal representations in an effective manner.

Comb Convolution for Efficient Convolutional Architecture

This paper presents a novel convolutional operator, namely comb convolution, to exploit the intra-channel sparse relationship among neurons, and demonstrates that by simply replacing standard convolutions with comb convolutions on state-of-the-art CNN architectures, it can achieve 50% FLOPs reduction while still maintaining the accuracy.

Holistic CNN Compression via Low-Rank Decomposition with Knowledge Transfer

This paper introduces a holistic CNN compression framework, termed LRDKT, which works throughout both convolutional and fully-connected layers, and introduces a novel knowledge transfer (KT) based training scheme, which has superior performance gains over the state-of-the-art methods.

Log-DenseNet: How to Sparsify a DenseNet

A connection template, Log-DenseNet is proposed, which, in comparison to DenseNet, only slightly increases the backpropagation distances among layers from 1 to ($1 + \log_2 L$), but uses only $L\log-2 L total connections instead of $O(L^2)$.

Variable batch size across layers for efficient prediction on CNNs

A dynamic program (DP) based algorithm that takes inference time and memory required by different layers of the network as input, and computes the optimal batch sizes for each layer depending on the available resources (RAM, storage space etc.).
...

References

SHOWING 1-10 OF 46 REFERENCES

Training CNNs with Low-Rank Filters for Efficient Image Classification

A new method for creating computationally efficient convolutional neural networks (CNNs) by using low-rank representations of Convolutional filters rather than approximating filters in previously-trained networks with more efficient versions, which shows similar or higher accuracy than conventional CNNs with much less compute.

Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition

A simple two-step approach for speeding up convolution layers within large convolutional neural networks based on tensor decomposition and discriminative fine-tuning is proposed, leading to higher obtained CPU speedups at the cost of lower accuracy drops for the smaller of the two networks.

Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications

A simple and effective scheme to compress the entire CNN, called one-shot whole network compression, which addresses the important implementation level issue on 1?1 convolution, which is a key operation of inception module of GoogLeNet as well as CNNs compressed by the proposed scheme.

Speeding up Convolutional Neural Networks with Low Rank Expansions

Two simple schemes for drastically speeding up convolutional neural networks are presented, achieved by exploiting cross-channel or filter redundancy to construct a low rank basis of filters that are rank-1 in the spatial domain.

Spectral Representations for Convolutional Neural Networks

This work proposes spectral pooling, which performs dimensionality reduction by truncating the representation in the frequency domain, and demonstrates the effectiveness of complex-coefficient spectral parameterization of convolutional filters.

Aggregated Residual Transformations for Deep Neural Networks

On the ImageNet-1K dataset, it is empirically show that even under the restricted condition of maintaining complexity, increasing cardinality is able to improve classification accuracy and is more effective than going deeper or wider when the authors increase the capacity.

Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation

Using large state-of-the-art models, this work demonstrates speedups of convolutional layers on both CPU and GPU by a factor of 2 x, while keeping the accuracy within 1% of the original model.

ImageNet classification with deep convolutional neural networks

A large, deep convolutional neural network was trained to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes and employed a recently developed regularization method called "dropout" that proved to be very effective.

Very Deep Convolutional Networks for Large-Scale Image Recognition

This work investigates the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting using an architecture with very small convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers.

Going deeper with convolutions

We propose a deep convolutional neural network architecture codenamed Inception that achieves the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition…