One-Shot Neural Architecture Search via Self-Evaluated Template Network

@article{Dong2019OneShotNA,
  title={One-Shot Neural Architecture Search via Self-Evaluated Template Network},
  author={Xuanyi Dong and Yezhou Yang},
  journal={2019 IEEE/CVF International Conference on Computer Vision (ICCV)},
  year={2019},
  pages={3680-3689}
}
  • Xuanyi Dong, Yezhou Yang
  • Published 1 October 2019
  • Computer Science
  • 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
Neural architecture search (NAS) aims to automate the search procedure of architecture instead of manual design. Even if recent NAS approaches finish the search within days, lengthy training is still required for a specific architecture candidate to get the parameters for its accurate evaluation. Recently one-shot NAS methods are proposed to largely squeeze the tedious training process by sharing parameters across candidates. In this way, the parameters for each candidate can be directly… 
A Comprehensive Survey of Neural Architecture Search
  • Pengzhen Ren, Yun Xiao, +4 authors Xin Wang
  • Computer Science
    ACM Comput. Surv.
  • 2021
TLDR
This survey provides a new perspective on Neural Architecture Search, beginning with an overview of the characteristics of the earliest NAS algorithms, summarizing the problems in these earlyNAS algorithms, and then providing solutions for subsequent related research work.
Delve into the Performance Degradation of Differentiable Architecture Search
TLDR
It is conjecture that the performance of DARTS does not depend on the well-trained supernet weights and it is argued that the architecture parameters should be trained by the gradients which are obtained in the early stage rather than the final stage of training.
Fine-Tuning DARTS for Image Classification
TLDR
This work proposes to fine-tune DARTS using fixed operations as they are independent of these approximations, and offers a good trade-off between the number of parameters and classification accuracy.
Rapid Neural Architecture Search by Learning to Generate Graphs from Datasets
TLDR
The proposed MetaD2A (Meta Dataset-to-Architecture) model can stochastically generate graphs from a given set (dataset) via a cross-modal latent space learned with amortized meta-learning and also proposes a meta-performance predictor to estimate and select the best architecture without direct training on target datasets.
A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
TLDR
This survey provides a new perspective on the NAS starting with an overview of the characteristics of the earliest NAS algorithms, summarizing the problems in these earlyNAS algorithms, and then giving solutions for subsequent related research work.
NATS-Bench: Benchmarking NAS algorithms for Architecture Topology and Size
TLDR
NTS-Bench is proposed, a unified benchmark on searching for both topology and size, for (almost) any up-to-date algorithm and shows the versatility of NATS-Bench by benchmarking 13 recent state-of-the-art NAS algorithms.
RADARS: Memory Efficient Reinforcement Learning Aided Differentiable Neural Architecture Search
  • Zheyu Yan, Weiwen Jiang, X. Hu, Yiyu Shi
  • Computer Science
    ArXiv
  • 2021
TLDR
To the best of the authors’ knowledge, this is the first DNAS framework that can handle large search spaces with bounded memory usage and RADARS is presented, a scalable RL aided DNAS Framework that can explore largeSearch spaces in a fast and memory-efficient manner.
NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search
TLDR
This work proposes an extension to NAS-bench-101: NAS-Bench-201 with a different search space, results on multiple datasets, and more diagnostic information, which provides additional diagnostic information such as fine-grained loss and accuracy, which can give inspirations to new designs of NAS algorithms.
PRODUCIBLE NEURAL ARCHITECTURE SEARCH
Neural architecture search (NAS) has achieved breakthrough success in a great number of applications in the past few years. It could be time to take a step back and analyze the good and bad aspects
A Deep Neural Networks ensemble workflow from hyperparameter search to inference leveraging GPU clusters
TLDR
This work proposes a new AutoML to build a larger library of accurate and diverse individual models to then construct ensembles, and proposes a novel algorithm to optimize the inference of the DNNs ensemble in a GPU cluster based on allocation optimization.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 45 REFERENCES
Learning Multiple Layers of Features from Tiny Images
TLDR
It is shown how to train a multi-layer generative model that learns to extract meaningful features which resemble those found in the human visual cortex, using a novel parallelization algorithm to distribute the work among multiple machines connected on a network.
DARTS: Differentiable Architecture Search
TLDR
The proposed algorithm excels in discovering high-performance convolutional architectures for image classification and recurrent architectures for language modeling, while being orders of magnitude faster than state-of-the-art non-differentiable techniques.
Learning Transferable Architectures for Scalable Image Recognition
TLDR
This paper proposes to search for an architectural building block on a small dataset and then transfer the block to a larger dataset and introduces a new regularization technique called ScheduledDropPath that significantly improves generalization in the NASNet models.
Progressive Neural Architecture Search
We propose a new method for learning the structure of convolutional neural networks (CNNs) that is more efficient than recent state-of-the-art methods based on reinforcement learning and evolutionary
SMASH: One-Shot Model Architecture Search through HyperNetworks
TLDR
A technique to accelerate architecture selection by learning an auxiliary HyperNet that generates the weights of a main model conditioned on that model's architecture is proposed, achieving competitive performance with similarly-sized hand-designed networks.
Graph HyperNetworks for Neural Architecture Search
TLDR
The GHN is proposed to amortize the search cost: given an architecture, it directly generates the weights by running inference on a graph neural network, which can predict network performance more accurately than regular hypernetworks and premature early stopping.
Efficient Neural Architecture Search via Parameter Sharing
TLDR
Efficient Neural Architecture Search is a fast and inexpensive approach for automatic model design that establishes a new state-of-the-art among all methods without post-training processing and delivers strong empirical performances using much fewer GPU-hours.
Neural Architecture Search with Reinforcement Learning
TLDR
This paper uses a recurrent network to generate the model descriptions of neural networks and trains this RNN with reinforcement learning to maximize the expected accuracy of the generated architectures on a validation set.
Densely Connected Search Space for More Flexible Neural Architecture Search
TLDR
This paper proposes to search block counts and block widths by designing a densely connected search space, i.e., DenseNAS, represented as a dense super network, which is built upon the designed routing blocks.
NAS-Bench-101: Towards Reproducible Neural Architecture Search
TLDR
This work introduces NAS-Bench-101, the first public architecture dataset for NAS research, which allows researchers to evaluate the quality of a diverse range of models in milliseconds by querying the pre-computed dataset.
...
1
2
3
4
5
...