AdvantageNAS: Efficient Neural Architecture Search with Credit Assignment

@inproceedings{Sato2020AdvantageNASEN,
  title={AdvantageNAS: Efficient Neural Architecture Search with Credit Assignment},
  author={Reimi Sato and Jun Sakuma and Youhei Akimoto},
  booktitle={AAAI Conference on Artificial Intelligence},
  year={2020}
}
Neural architecture search (NAS) is an approach for automatically designing a neural network architecture without human effort or expert knowledge. However, the high computational cost of NAS limits its use in commercial applications. Two recent NAS paradigms, namely one-shot and sparse propagation, which reduce the time and space complexities, respectively, provide clues for solving this problem. In this paper, we propose a novel search strategy for one-shot and sparse propagation NAS, namely… 
[PDF] Semantic Reader
1 Citations

Figures and Tables from this paper

One Citation

GPNAS: A Neural Network Architecture Search Framework Based on Graphical Predictor

A framework to decouple network structure from operator search space, and use two BOHBs to search alternatively, and introduces an activation function and an initialization method domain to improve the generalization ability of the child model.

References

SHOWING 1-10 OF 25 REFERENCES

Efficient Neural Architecture Search via Parameter Sharing

Efficient Neural Architecture Search is a fast and inexpensive approach for automatic model design that establishes a new state-of-the-art among all methods without post-training processing and delivers strong empirical performances using much fewer GPU-hours.

Efficient Neural Architecture Search via Proximal Iterations

This work reformulates the search process as an optimization problem with a discrete constraint on architectures and a regularizer on model complexity and proposes an efficient algorithm inspired by proximal iterations for optimization that is not only much faster than existing differentiable search methods, but also can find better architectures and balance the model complexity.

SNAS: Stochastic Neural Architecture Search

It is proved that this search gradient optimizes the same objective as reinforcement-learning-based NAS, but assigns credits to structural decisions more efficiently, and is further augmented with locally decomposable reward to enforce a resource-efficient constraint.

Probabilistic Neural Architecture Search

A Probabilistic approach to neural ARchitecture SEarCh (PARSEC) that drastically reduces memory requirements while maintaining state-of-the-art computational complexity, making it possible to directly search over more complex architectures and larger datasets.

NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search

This work proposes an extension to NAS-bench-101: NAS-Bench-201 with a different search space, results on multiple datasets, and more diagnostic information, which provides additional diagnostic information such as fine-grained loss and accuracy, which can give inspirations to new designs of NAS algorithms.

ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware

ProxylessNAS is presented, which can directly learn the architectures for large-scale target tasks and target hardware platforms and apply ProxylessNAS to specialize neural architectures for hardware with direct hardware metrics (e.g. latency) and provide insights for efficient CNN architecture design.

Searching for a Robust Neural Architecture in Four GPU Hours

  • Xuanyi DongYezhou Yang
  • Computer Science
    2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2019
The approach can be trained in an end-to-end fashion by gradient descent, named Gradient-based search using Differentiable Architecture Sampler (GDAS), and the discovered model obtains a test error of 2.82% with only 2.5M parameters, which is on par with the state-of-the-art.

Neural Architecture Search with Reinforcement Learning

This paper uses a recurrent network to generate the model descriptions of neural networks and trains this RNN with reinforcement learning to maximize the expected accuracy of the generated architectures on a validation set.

DARTS: Differentiable Architecture Search

The proposed algorithm excels in discovering high-performance convolutional architectures for image classification and recurrent architectures for language modeling, while being orders of magnitude faster than state-of-the-art non-differentiable techniques.

NAS evaluation is frustratingly hard

This work proposes using a method’s relative improvement over the randomly sampled average architecture, which effectively removes advantages arising from expertly engineered search spaces or training protocols to overcome the hurdle of comparing methods with different search spaces.