Corpus ID: 209516112

NAS evaluation is frustratingly hard

@article{Yang2020NASEI,
  title={NAS evaluation is frustratingly hard},
  author={Antoine Yang and Pedro M. Esperança and Fabio Maria Carlucci},
  journal={ArXiv},
  year={2020},
  volume={abs/1912.12522}
}
Neural Architecture Search (NAS) is an exciting new field which promises to be as much as a game-changer as Convolutional Neural Networks were in 2012. Despite many great works leading to substantial improvements on a variety of tasks, comparison between different methods is still very much an open issue. While most algorithms are tested on the same datasets, there is no shared experimental protocol followed by all. As such, and due to the under-use of ablation studies, there is a lack of… Expand
NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
TLDR
A general framework for one-shot NAS that can be instantiated to many recently-introduced variants and a general benchmarking framework that draws on the recent large-scale tabular benchmark NAS-Bench-101 for cheap anytime evaluations of one- shot NAS methods are introduced. Expand
NASLib: A Modular and Flexible Neural Architecture Search Library
TLDR
NASLib, a NAS library built upon PyTorch, offers high-level abstractions for designing and reusing search spaces, interfaces to benchmarks and evaluation pipelines, enabling the implementation and extension of state-of-theart NAS methods with a few lines of code. Expand
ONE-SHOT NEURAL ARCHITECTURE SEARCH
One-shot neural architecture search (NAS) has played a crucial role in making NAS methods computationally feasible in practice. Nevertheless, there is still a lack of understanding on how theseExpand
Revisiting Neural Architecture Search
TLDR
This paper revisits the fundamental approach to NAS and proposes a novel approach called ReNAS that can search for the complete neural network without much human effort and is a step closer towards AutoML-nirvana. Expand
Block-Wisely Supervised Neural Architecture Search With Knowledge Distillation
TLDR
This work proposes to modularize the large search space of NAS into blocks to ensure that the potential candidate architectures are fully trained, and distill the neural architecture (DNA) knowledge from a teacher model to supervise the block-wise architecture search, which significantly improves the effectiveness of NAS. Expand
TRANSNAS-BENCH-101: IMPROVING TRANSFERRA-
  • 2020
Recent breakthroughs of Neural Architecture Search (NAS) are extending the field’s research scope towards a broader range of vision tasks and more diversified search spaces. While existing NASExpand
EPE-NAS: Efficient Performance Estimation Without Training for Neural Architecture Search
TLDR
EPE-NAS is an efficient performance estimation strategy, that mitigates the problem of evaluating networks, by scoring untrained networks and creating a correlation with their trained performance, and is agnostic to the search method, making it easy to integrate into almost any NAS method. Expand
TransNAS-Bench-101: Improving Transferability and Generalizability of Cross-Task Neural Architecture Search
TLDR
This work proposes TransNAS-Bench-101, a benchmark dataset containing network performance across seven tasks, covering classification, regression, pixel-level prediction, and self-supervised tasks, and explores two fundamentally different types of search space: cell-level search space and macro- level search space. Expand
Multi-fidelity Neural Architecture Search with Knowledge Distillation
TLDR
This paper proposes to improve low-fidelity evaluations of neural architectures by using a knowledge distillation and shows that training on the small part of a dataset with such a modified loss function leads to a better selection of neural architecture than training with a logistic loss. Expand
Rethinking Neural Operations for Diverse Tasks
TLDR
This work introduces a search space of neural operations called XD-Operations that mimic the inductive bias of standard multichannel convolutions while being much more expressive: it is proved that XD-operations include many named operations across several application areas. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 41 REFERENCES
NAS-Bench-101: Towards Reproducible Neural Architecture Search
TLDR
This work introduces NAS-Bench-101, the first public architecture dataset for NAS research, which allows researchers to evaluate the quality of a diverse range of models in milliseconds by querying the pre-computed dataset. Expand
sharpDARTS: Faster and More Accurate Differentiable Architecture Search
TLDR
This work proposes Differentiable Hyperparameter Grid Search and the HyperCuboid search space, which are representations designed to leverage DARTS for more general parameter optimization, and proposes Max-W regularization to solve this problem. Expand
Evaluating the Search Phase of Neural Architecture Search
TLDR
This paper finds that on average, the state-of-the-art NAS algorithms perform similarly to the random policy; the widely-used weight sharing strategy degrades the ranking of the NAS candidates to the point of not reflecting their true performance, thus reducing the effectiveness of the search process. Expand
Deep Networks with Stochastic Depth
TLDR
Stochastic depth is proposed, a training procedure that enables the seemingly contradictory setup to train short networks and use deep networks at test time and reduces training time substantially and improves the test error significantly on almost all data sets that were used for evaluation. Expand
Large-Scale Evolution of Image Classifiers
TLDR
It is shown that it is now possible to evolve models with accuracies within the range of those published in the last year, starting from trivial initial conditions and reaching accuracies of 94.6% and 77.0%, respectively. Expand
Neural Architecture Search with Reinforcement Learning
TLDR
This paper uses a recurrent network to generate the model descriptions of neural networks and trains this RNN with reinforcement learning to maximize the expected accuracy of the generated architectures on a validation set. Expand
Improved Regularization of Convolutional Neural Networks with Cutout
TLDR
This paper shows that the simple regularization technique of randomly masking out square regions of input during training, which is called cutout, can be used to improve the robustness and overall performance of convolutional neural networks. Expand
Progressive Differentiable Architecture Search: Bridging the Depth Gap Between Search and Evaluation
TLDR
This paper presents an efficient algorithm which allows the depth of searched architectures to grow gradually during the training procedure, and solves two issues, namely, heavier computational overheads and weaker search stability, which are solved using search space approximation and regularization. Expand
DARTS: Differentiable Architecture Search
TLDR
The proposed algorithm excels in discovering high-performance convolutional architectures for image classification and recurrent architectures for language modeling, while being orders of magnitude faster than state-of-the-art non-differentiable techniques. Expand
Efficient Neural Architecture Search via Parameter Sharing
TLDR
Efficient Neural Architecture Search is a fast and inexpensive approach for automatic model design that establishes a new state-of-the-art among all methods without post-training processing and delivers strong empirical performances using much fewer GPU-hours. Expand
...
1
2
3
4
5
...