• Corpus ID: 85542624

Network Slimming by Slimmable Networks: Towards One-Shot Architecture Search for Channel Numbers

@article{Yu2019NetworkSB,
  title={Network Slimming by Slimmable Networks: Towards One-Shot Architecture Search for Channel Numbers},
  author={Jiahui Yu and Thomas S. Huang},
  journal={ArXiv},
  year={2019},
  volume={abs/1903.11728}
}
We study how to set channel numbers in a neural network to achieve better accuracy under constrained resources (e.g., FLOPs, latency, memory footprint or model size). A simple and one-shot solution, named AutoSlim, is presented. Instead of training many network samples and searching with reinforcement learning, we train a single slimmable network to approximate the network accuracy of different channel configurations. We then iteratively evaluate the trained slimmable model and greedily slim… 

Figures and Tables from this paper

BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models
TLDR
The proposed BigNAS, an approach that challenges the conventional wisdom that post-processing of the weights is necessary to get good prediction accuracies, is proposed, able to train a single set of shared weights on ImageNet and use these weights to obtain child models whose sizes range from 200 to 1000 MFLOPs.
Universally Slimmable Networks and Improved Training Techniques
TLDR
This work proposes a systematic approach to train universally slimmable networks (US-Nets), extending slimmables to execute at arbitrary width, and generalizing to networks both with and without batch normalization layers, and opens up the possibility to directly evaluate FLOPs-Accuracy spectrum of network architectures.
Network Adjustment: Channel Search Guided by FLOPs Utilization Ratio
TLDR
A new framework named network adjustment is presented, which considers network accuracy as a function ofFLOPs, so that under each network configuration, one can estimate the FLOPs utilization ratio (FUR) for each layer and use it to determine whether to increase or decrease the number of channels on the layer.
Switchable Precision Neural Networks
TLDR
This paper proposes a flexible quantization strategy, termed Switchable Precision neural Networks (SP-Nets), to train a shared network capable of operating at multiple quantization levels, which can adjust its precision on the fly according to instant memory, latency, power consumption and accuracy demands.
Network Pruning via Transformable Architecture Search
TLDR
Neural architecture search is proposed to apply neural architecture search to search directly for a network with flexible channel and layer sizes to break the structure limitation of the pruned networks.
AdaBits: Neural Network Quantization With Adaptive Bit-Widths
TLDR
It is demonstrated that bit-width of weights and activations is a new option for adaptively executable deep neural networks, offering a distinct opportunity for improved accuracy-efficiency trade-off as well as instant adaptation according to the platform constraints in real-world applications.
Improved Techniques for Quantizing Deep Networks with Adaptive Bit-Widths
TLDR
This paper presents two novel techniques for quantizing deep neural networks with adaptive bit-widths of weights and activations and proposes a collaborative strategy to choose a high-precision “teacher” for transferring knowledge to the low-pre precision “student” while jointly optimizing the model with all bit- widths.
Out-of-the-box channel pruned networks
TLDR
A Reinforcement Learning (RL) policy-based search algorithm with a direct objective of finding transferable layer-wise pruning profiles using many models for the same architecture is developed and results show that the transferred RL-based profiles are as good or better than best profiles found on the original dataset via exhaustive search.
Understanding and Improving One-shot Neural Architecture Optimization
TLDR
This work empirically investigates several main factors that lead to the gaps and so weak ranking correlation between architectures under one-shot training and the ones under stand-alone complete training and proposes NAO-V2 to alleviate such gaps.
Deep Model Compression via Filter Auto-sampling
TLDR
This work presents a novel autosampling method that is applicable to both 1D and 2D CNNs with significant performance improvement over WSNet and even outperforms some neural architecture search (NAS) based methods such as AMC and MNasNet.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 59 REFERENCES
SMASH: One-Shot Model Architecture Search through HyperNetworks
TLDR
A technique to accelerate architecture selection by learning an auxiliary HyperNet that generates the weights of a main model conditioned on that model's architecture is proposed, achieving competitive performance with similarly-sized hand-designed networks.
Slimmable Neural Networks
TLDR
This work presents a simple and general method to train a single neural network executable at different widths, permitting instant and adaptive accuracy-efficiency trade-offs at runtime, and demonstrates better performance of slimmable models compared with individual ones across a wide range of applications.
Universally Slimmable Networks and Improved Training Techniques
TLDR
This work proposes a systematic approach to train universally slimmable networks (US-Nets), extending slimmables to execute at arbitrary width, and generalizing to networks both with and without batch normalization layers, and opens up the possibility to directly evaluate FLOPs-Accuracy spectrum of network architectures.
SNIP: Single-shot Network Pruning based on Connection Sensitivity
TLDR
This work presents a new approach that prunes a given network once at initialization prior to training, and introduces a saliency criterion based on connection sensitivity that identifies structurally important connections in the network for the given task.
MnasNet: Platform-Aware Neural Architecture Search for Mobile
TLDR
An automated mobile neural architecture search (MNAS) approach, which explicitly incorporate model latency into the main objective so that the search can identify a model that achieves a good trade-off between accuracy and latency.
Learning Efficient Convolutional Networks through Network Slimming
TLDR
The approach is called network slimming, which takes wide and large networks as input models, but during training insignificant channels are automatically identified and pruned afterwards, yielding thin and compact models with comparable accuracy.
Rethinking the Inception Architecture for Computer Vision
TLDR
This work is exploring ways to scale up networks in ways that aim at utilizing the added computation as efficiently as possible by suitably factorized convolutions and aggressive regularization.
SkipNet: Learning Dynamic Routing in Convolutional Networks
TLDR
This work introduces SkipNet, a modified residual network, that uses a gating network to selectively skip convolutional blocks based on the activations of the previous layer, and proposes a hybrid learning algorithm that combines supervised learning and reinforcement learning to address the challenges of non-differentiable skipping decisions.
Understanding and Simplifying One-Shot Architecture Search
TLDR
With careful experimental analysis, it is shown that it is possible to efficiently identify promising architectures from a complex search space without either hypernetworks or reinforcement learning controllers.
Efficient Neural Architecture Search via Parameter Sharing
TLDR
Efficient Neural Architecture Search is a fast and inexpensive approach for automatic model design that establishes a new state-of-the-art among all methods without post-training processing and delivers strong empirical performances using much fewer GPU-hours.
...
1
2
3
4
5
...