NetAdaptV2: Efficient Neural Architecture Search with Fast Super-Network Training and Architecture Optimization

  title={NetAdaptV2: Efficient Neural Architecture Search with Fast Super-Network Training and Architecture Optimization},
  author={Tien-Ju Yang and Yi Liao and Vivienne Sze},
  journal={2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  • Tien-Ju Yang, Yi Liao, V. Sze
  • Published 31 March 2021
  • Computer Science
  • 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Neural architecture search (NAS) typically consists of three main steps: training a super-network, training and evaluating sampled deep neural networks (DNNs), and training the discovered DNN. Most of the existing efforts speed up some steps at the cost of a significant slowdown of other steps or sacrificing the support of non-differentiable search metrics. The unbalanced reduction in the time spent per step limits the total search time reduction, and the inability to support non-differentiable… 
Searching for Efficient Multi-Stage Vision Transformers
This work proposes ViT-ResNAS, an efficient multi-stage ViT architecture designed with neural architecture search (NAS) that achieves better accuracy-MACs and accuracy-throughput trade-offs than the original DeiT and other strong baselines of ViT.
Partial Variable Training for Efficient On-Device Federated Learning
Partial Variable Training (PVT) is proposed, a novel method that only trains a small subset of variables on edge devices to reduce memory usage and communication cost and shows that network accuracy can be maintained by utilizing more local training steps and devices, which is favorable for FL involving a large population of devices.


BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models
The proposed BigNAS, an approach that challenges the conventional wisdom that post-processing of the weights is necessary to get good prediction accuracies, is proposed, able to train a single set of shared weights on ImageNet and use these weights to obtain child models whose sizes range from 200 to 1000 MFLOPs.
Efficient Neural Architecture Search via Parameter Sharing
Efficient Neural Architecture Search is a fast and inexpensive approach for automatic model design that establishes a new state-of-the-art among all methods without post-training processing and delivers strong empirical performances using much fewer GPU-hours.
ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
ProxylessNAS is presented, which can directly learn the architectures for large-scale target tasks and target hardware platforms and apply ProxylessNAS to specialize neural architectures for hardware with direct hardware metrics (e.g. latency) and provide insights for efficient CNN architecture design.
AutoSlim: Towards One-Shot Architecture Search for Channel Numbers
A simple and one-shot solution to set channel numbers in a neural network to achieve better accuracy under constrained resources (e.g., FLOPs, latency, memory footprint or model size) is presented.
PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
This paper presents a novel approach, namely Partially-Connected DARTS, by sampling a small part of super-net to reduce the redundancy in exploring the network space, thereby performing a more efficient search without comprising the performance.
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
A new scaling method is proposed that uniformly scales all dimensions of depth/width/resolution using a simple yet highly effective compound coefficient and is demonstrated the effectiveness of this method on scaling up MobileNets and ResNet.
Single-Path Mobile AutoML: Efficient ConvNet Design and NAS Hyperparameter Optimization
This article proposes a novel differentiable NAS formulation, namely Single-Path NAS, that uses one single-path over-parameterized ConvNet to encode all architectural decisions based on shared convolutional kernel parameters, hence drastically decreasing the search overhead and outperforming existing mobile NAS methods.
AtomNAS: Fine-Grained End-to-End Neural Architecture Search
A fine-grained search space comprised of atomic blocks, a minimal search unit much smaller than the ones used in recent NAS algorithms is proposed, which achieves state-of-the-art performance under several FLOPS configurations on ImageNet with a negligible searching cost.
Deep Networks with Stochastic Depth
Stochastic depth is proposed, a training procedure that enables the seemingly contradictory setup to train short networks and use deep networks at test time and reduces training time substantially and improves the test error significantly on almost all data sets that were used for evaluation.
FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions
This work proposes a memory and computationally efficient DNAS variant, DMaskingNAS, that expands the search space by up to 10^14x over conventional DNAS, supporting searches over spatial and channel dimensions that are otherwise prohibitively expensive: input resolution and number of filters.