Single Path One-Shot Neural Architecture Search with Uniform Sampling

@inproceedings{Guo2019SinglePO,
  title={Single Path One-Shot Neural Architecture Search with Uniform Sampling},
  author={Zichao Guo and Xiangyu Zhang and Haoyuan Mu and Wen Heng and Zechun Liu and Yichen Wei and Jian Sun},
  booktitle={European Conference on Computer Vision},
  year={2019}
}
One-shot method is a powerful Neural Architecture Search (NAS) framework, but its training is non-trivial and it is difficult to achieve competitive results on large scale datasets like ImageNet. [] Key Method Once we have a trained supernet, we apply an evolutionary algorithm to efficiently search the best-performing architectures without any fine tuning. Comprehensive experiments verify that our approach is flexible and effective. It is easy to train and fast to search. It effortlessly supports complex…

MixPath: A Unified Approach for One-shot Neural Architecture Search

This paper discovers that in the studied search space, feature vectors summed from multiple paths are nearly multiples of those from a single path, which perturbs supernet training and its ranking ability, and proposes a novel mechanism called Shadow Batch Normalization (SBN) to regularize the disparate feature statistics.

Progressive Automatic Design of Search Space for One-Shot Neural Architecture Search

This paper formulate a progressive search strategy based on operation pruning and build a layer-wise operation search space, named PAD-NAS, which can automatically design the operations for each layer and achieve a trade-off between search space quality and model diversity.

Few-shot Neural Architecture Search

This work proposes few-shot NAS that explores the choice of using multiple super-nets: each super-net is pre-trained to be in charge of a sub-region of the search space, which reduces the prediction error of eachsuper-net.

Generalizing Few-Shot NAS with Gradient Matching

A gradient matching score (GM) that leverages gradient information at the shared weight for making informed splitting decisions is proposed that significantly outperforms its Few-Shot counterparts while surpassing previous comparable methods in terms of the accuracy of derived architectures.

Layer-Wisely Supervised Learning For One-Shot Neural Architecture Search

This paper proposes to modularize the large weight-sharing space of one-shot NAS into layers by introducing layer-wisely supervised learning, finding that greedy layer- wise learning that learns each layer separately with a local objective hurts super-net performance as well as ranking correlation.

Balanced One-shot Neural Architecture Optimization

Balanced NAO is proposed where balanced training of the supernet is introduced during the search procedure to encourage more updates for large architectures than small architectures by sampling architectures in proportion to their model sizes.

Improving One-Shot NAS by Suppressing the Posterior Fading

This paper analyzes existing weight sharing one-shot NAS approaches from a Bayesian point of view and identifies the Posterior Fading problem, which compromises the effectiveness of shared weights, and presents a novel approach to guide the parameter posterior towards its true distribution.

Powering One-shot Topological NAS with Stabilized Share-parameter Proxy

The difficulties for architecture searching in such a complex space has been eliminated by the proposed stabilized share-parameter proxy, which employs Stochastic Gradient Langevin Dynamics to enable fast shared parameter sampling, so as to achieve stabilized measurement of architecture performance even in search space with complex topological structures.

GreedyNAS: Towards Fast One-Shot NAS With Greedy Supernet

This paper proposes a multi-path sampling strategy with rejection, and greedily filter the weak paths to ease the burden of supernet by encouraging it to focus more on evaluation of those potentially-good ones, which are identified using a surrogate portion of validation data.

Understanding and Improving One-shot Neural Architecture Optimization

This work empirically investigates several main factors that lead to the gaps and so weak ranking correlation between architectures under one-shot training and the ones under stand-alone complete training and proposes NAO-V2 to alleviate such gaps.
...

References

SHOWING 1-10 OF 48 REFERENCES

You Only Search Once: Single Shot Neural Architecture Search via Direct Sparse Optimization

The motivation behind DSO-NAS is to address the task in the view of model pruning, and it enjoys both advantages of differentiability and efficiency, therefore it can be directly applied to large datasets like ImageNet and tasks beyond classification.

Single-Path NAS: Designing Hardware-Efficient ConvNets in less than 4 Hours

This work proposes Single-Path NAS, a novel differentiable NAS method for designing hardware-efficient ConvNets in less than 4 hours, and uses one single-path over-parameterized ConvNet to encode all architectural decisions with shared convolutional kernel parameters, hence drastically decreasing the number of trainable parameters and the search cost down to few epochs.

BlockQNN: Efficient Block-Wise Neural Network Architecture Generation

This paper provides a block-wise network generation pipeline called BlockQNN which automatically builds high-performance networks using the Q-Learning paradigm with epsilon-greedy exploration strategy and proposes a distributed asynchronous framework and an early stop strategy.

Practical Block-Wise Neural Network Architecture Generation

A block-wise network generation pipeline called BlockQNN which automatically builds high-performance networks using the Q-Learning paradigm with epsilon-greedy exploration strategy and offers tremendous reduction of the search space in designing networks which only spends 3 days with 32 GPUs.

Learning Transferable Architectures for Scalable Image Recognition

This paper proposes to search for an architectural building block on a small dataset and then transfer the block to a larger dataset and introduces a new regularization technique called ScheduledDropPath that significantly improves generalization in the NASNet models.

SMASH: One-Shot Model Architecture Search through HyperNetworks

A technique to accelerate architecture selection by learning an auxiliary HyperNet that generates the weights of a main model conditioned on that model's architecture is proposed, achieving competitive performance with similarly-sized hand-designed networks.

MnasNet: Platform-Aware Neural Architecture Search for Mobile

An automated mobile neural architecture search (MNAS) approach, which explicitly incorporate model latency into the main objective so that the search can identify a model that achieves a good trade-off between accuracy and latency.

Efficient Neural Architecture Search via Proximal Iterations

This work reformulates the search process as an optimization problem with a discrete constraint on architectures and a regularizer on model complexity and proposes an efficient algorithm inspired by proximal iterations for optimization that is not only much faster than existing differentiable search methods, but also can find better architectures and balance the model complexity.

Learning Time/Memory-Efficient Deep Architectures with Budgeted Super Networks

A novel family of models called Budgeted Super Networks (BSN) is proposed, learned using gradient descent techniques applied on a budgeted learning objective function which integrates a maximum authorized cost, while making no assumption on the nature of this cost.

Understanding and Simplifying One-Shot Architecture Search

With careful experimental analysis, it is shown that it is possible to efficiently identify promising architectures from a complex search space without either hypernetworks or reinforcement learning controllers.