NetAdaptV2: Efficient Neural Architecture Search with Fast Super-Network Training and Architecture Optimization

@article{Yang2021NetAdaptV2EN,
  title={NetAdaptV2: Efficient Neural Architecture Search with Fast Super-Network Training and Architecture Optimization},
  author={Tien-Ju Yang and Yi Liao and Vivienne Sze},
  journal={2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2021},
  pages={2402-2411}
}
  • Tien-Ju Yang, Yi Liao, V. Sze
  • Published 31 March 2021
  • Computer Science
  • 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Neural architecture search (NAS) typically consists of three main steps: training a super-network, training and evaluating sampled deep neural networks (DNNs), and training the discovered DNN. Most of the existing efforts speed up some steps at the cost of a significant slowdown of other steps or sacrificing the support of non-differentiable search metrics. The unbalanced reduction in the time spent per step limits the total search time reduction, and the inability to support non-differentiable… Expand
Searching for Efficient Multi-Stage Vision Transformers
TLDR
This work proposes ViT-ResNAS, an efficient multi-stage ViT architecture designed with neural architecture search (NAS) that achieves better accuracy-MACs and accuracy-throughput trade-offs than the original DeiT and other strong baselines of ViT. Expand
Partial Variable Training for Efficient On-Device Federated Learning
TLDR
Partial Variable Training (PVT) is proposed, a novel method that only trains a small subset of variables on edge devices to reduce memory usage and communication cost and shows that network accuracy can be maintained by utilizing more local training steps and devices, which is favorable for FL involving a large population of devices. Expand

References

SHOWING 1-10 OF 42 REFERENCES
BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models
TLDR
The proposed BigNAS, an approach that challenges the conventional wisdom that post-processing of the weights is necessary to get good prediction accuracies, is proposed, able to train a single set of shared weights on ImageNet and use these weights to obtain child models whose sizes range from 200 to 1000 MFLOPs. Expand
Efficient Neural Architecture Search via Parameter Sharing
TLDR
Efficient Neural Architecture Search is a fast and inexpensive approach for automatic model design that establishes a new state-of-the-art among all methods without post-training processing and delivers strong empirical performances using much fewer GPU-hours. Expand
ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
TLDR
ProxylessNAS is presented, which can directly learn the architectures for large-scale target tasks and target hardware platforms and apply ProxylessNAS to specialize neural architectures for hardware with direct hardware metrics (e.g. latency) and provide insights for efficient CNN architecture design. Expand
AutoSlim: Towards One-Shot Architecture Search for Channel Numbers
TLDR
A simple and one-shot solution to set channel numbers in a neural network to achieve better accuracy under constrained resources (e.g., FLOPs, latency, memory footprint or model size) is presented. Expand
PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
TLDR
This paper presents a novel approach, namely Partially-Connected DARTS, by sampling a small part of super-net to reduce the redundancy in exploring the network space, thereby performing a more efficient search without comprising the performance. Expand
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
TLDR
A new scaling method is proposed that uniformly scales all dimensions of depth/width/resolution using a simple yet highly effective compound coefficient and is demonstrated the effectiveness of this method on scaling up MobileNets and ResNet. Expand
Single-Path Mobile AutoML: Efficient ConvNet Design and NAS Hyperparameter Optimization
TLDR
This article proposes a novel differentiable NAS formulation, namely Single-Path NAS, that uses one single-path over-parameterized ConvNet to encode all architectural decisions based on shared convolutional kernel parameters, hence drastically decreasing the search overhead and outperforming existing mobile NAS methods. Expand
AtomNAS: Fine-Grained End-to-End Neural Architecture Search
TLDR
A fine-grained search space comprised of atomic blocks, a minimal search unit much smaller than the ones used in recent NAS algorithms is proposed, which achieves state-of-the-art performance under several FLOPS configurations on ImageNet with a negligible searching cost. Expand
Deep Networks with Stochastic Depth
TLDR
Stochastic depth is proposed, a training procedure that enables the seemingly contradictory setup to train short networks and use deep networks at test time and reduces training time substantially and improves the test error significantly on almost all data sets that were used for evaluation. Expand
FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions
TLDR
This work proposes a memory and computationally efficient DNAS variant, DMaskingNAS, that expands the search space by up to 10^14x over conventional DNAS, supporting searches over spatial and channel dimensions that are otherwise prohibitively expensive: input resolution and number of filters. Expand
...
1
2
3
4
5
...