• Corpus ID: 102352813

ASAP: Architecture Search, Anneal and Prune

@inproceedings{Noy2019ASAPAS,
  title={ASAP: Architecture Search, Anneal and Prune},
  author={Asaf Noy and Niv Nayman and T. Ridnik and Nadav Zamir and Sivan Doveh and Itamar Friedman and Raja Giryes and Lihi Zelnik-Manor},
  booktitle={International Conference on Artificial Intelligence and Statistics},
  year={2019}
}
Automatic methods for Neural Architecture Search (NAS) have been shown to produce state-of-the-art network models. Yet, their main drawback is the computational complexity of the search process. As some primal methods optimized over a discrete search space, thousands of days of GPU were required for convergence. A recent approach is based on constructing a differentiable search space that enables gradient-based optimization, which reduces the search time to a few days. While successful, it… 
[PDF] Semantic Reader
88 Citations
Highly Influential Citations
10
Background Citations
34
Methods Citations
22
Results Citations
2

Figures and Tables from this paper

88 Citations

Efficient Neural Architecture Search via Proximal Iterations

This work reformulates the search process as an optimization problem with a discrete constraint on architectures and a regularizer on model complexity and proposes an efficient algorithm inspired by proximal iterations for optimization that is not only much faster than existing differentiable search methods, but also can find better architectures and balance the model complexity.

DARTS+: Improved Differentiable Architecture Search with Early Stopping

It is claimed that there exists overfitting in the optimization of DARTS, and a simple and effective algorithm is proposed, named "DARTS+", to avoid the collapse and improve the original DARts, by "early stopping" the search procedure when meeting a certain criterion.

VINNAS: Variational Inference-based Neural Network Architecture Search

The approach finds the optimal neural architecture by dropping out candidate operations in an over-parameterised supergraph using variational dropout with automatic relevance determination prior, which makes the algorithm gradually remove unnecessary operations and connections without risking mode collapse.

RAPDARTS: Resource-Aware Progressive Differentiable Architecture Search

A popular gradient-based NAS method is extended to support one or more resource costs, and in-depth analysis is performed on the discovery of architectures satisfying single-resource constraints for classification of CIFAR-10.

Geometry-Aware Gradient Algorithms for Neural Architecture Search

A geometry-aware framework is presented that exploits the underlying structure of this optimization to return sparse architectural parameters, leading to simple yet novel algorithms that enjoy fast convergence guarantees and achieve state-of-the-art accuracy on the latest NAS benchmarks in computer vision.

Differentiable Architecture Search Meets Network Pruning at Initialization: A More Reliable, Efficient, and Flexible Framework

This work customizes the NAS as a network pruning at initialization problem, and seeks to score the importance of candidate operations in differentiable NAS without any training, and proposes a novel framework called training free differentiable architecture search (FreeDARTS), which can outperform most NAS baselines in different search spaces.

Cyclic Differentiable Architecture Search

This work proposes new joint optimization objectives and a novel Cyclic Differentiable ARchiTecture Search framework, dubbed CDARTS, which enables the evolution of the architecture to fit the final evaluation network.

PR-DARTS: Pruning-Based Differentiable Architecture Search

This work proposes two novel PrunedConv and PrunedLinear operations that mitigate the problem of unstable gradients by regularizing the objective function of the pruned networks and outperforms those used in the state-of-the-art pruning networks on CIFAR-10 and ImageNet.

BINAS: Bilinear Interpretable Neural Architecture Search

A Bilinear Interpretable approach for constrained Neural Architecture Search (BINAS) that is based on an accurate and simple bilinear formulation of both an accuracy estimator and the expected resource requirement, jointly with a scalable search method with theoretical guarantees is introduced.

IQNAS: Interpretable Integer Quadratic Programming Neural Architecture Search

This work introduces Interpretable Integer Quadratic programming Neural Architecture Search (IQNAS), that is based on an accurate and simple quadratic formulation of both the accuracy predictor and the expected resource requirement, together with a scalable search method with theoretical guarantees.
...

References

SHOWING 1-10 OF 52 REFERENCES

sharpDARTS: Faster and More Accurate Differentiable Architecture Search

This work proposes Differentiable Hyperparameter Grid Search and the HyperCuboid search space, which are representations designed to leverage DARTS for more general parameter optimization, and proposes Max-W regularization to solve this problem.

DARTS+: Improved Differentiable Architecture Search with Early Stopping

It is claimed that there exists overfitting in the optimization of DARTS, and a simple and effective algorithm is proposed, named "DARTS+", to avoid the collapse and improve the original DARts, by "early stopping" the search procedure when meeting a certain criterion.

Progressive Differentiable Architecture Search: Bridging the Depth Gap Between Search and Evaluation

This paper presents an efficient algorithm which allows the depth of searched architectures to grow gradually during the training procedure, and solves two issues, namely, heavier computational overheads and weaker search stability, which are solved using search space approximation and regularization.

Probabilistic Neural Architecture Search

A Probabilistic approach to neural ARchitecture SEarCh (PARSEC) that drastically reduces memory requirements while maintaining state-of-the-art computational complexity, making it possible to directly search over more complex architectures and larger datasets.

SNAS: Stochastic Neural Architecture Search

It is proved that this search gradient optimizes the same objective as reinforcement-learning-based NAS, but assigns credits to structural decisions more efficiently, and is further augmented with locally decomposable reward to enforce a resource-efficient constraint.

You Only Search Once: Single Shot Neural Architecture Search via Direct Sparse Optimization

The motivation behind DSO-NAS is to address the task in the view of model pruning, and it enjoys both advantages of differentiability and efficiency, therefore it can be directly applied to large datasets like ImageNet and tasks beyond classification.

Efficient Neural Architecture Search via Parameter Sharing

Efficient Neural Architecture Search is a fast and inexpensive approach for automatic model design that establishes a new state-of-the-art among all methods without post-training processing and delivers strong empirical performances using much fewer GPU-hours.

Regularized Evolution for Image Classifier Architecture Search

This work evolves an image classifier---AmoebaNet-A---that surpasses hand-designs for the first time and gives evidence that evolution can obtain results faster with the same hardware, especially at the earlier stages of the search.

XNAS: Neural Architecture Search with Expert Advice

This paper introduces a novel optimization method for differential neural architecture search, based on the theory of prediction with expert advice, that achieves an optimal worst-case regret bound and suggests the use of multiple learning-rates,based on the amount of information carried by the backward gradients.

Neural Architecture Optimization

Experiments show that the architecture discovered by this simple and efficient method to automatic neural architecture design based on continuous optimization is very competitive for image classification task on CIFAR-10 and language modeling task on PTB, outperforming or on par with the best results of previous architecture search methods with a significantly reduction of computational resources.
...