Corpus ID: 3638969

Efficient Neural Architecture Search via Parameter Sharing

@inproceedings{Pham2018EfficientNA,
  title={Efficient Neural Architecture Search via Parameter Sharing},
  author={Hieu Pham and M. Y. Guan and Barret Zoph and Quoc V. Le and Jeff Dean},
  booktitle={ICML},
  year={2018}
}
We propose Efficient Neural Architecture Search (ENAS), a fast and inexpensive approach for automatic model design. In ENAS, a controller learns to discover neural network architectures by searching for an optimal subgraph within a large computational graph. The controller is trained with policy gradient to select a subgraph that maximizes the expected reward on the validation set. Meanwhile the model corresponding to the selected subgraph is trained to minimize a canonical cross entropy loss… Expand
ADWPNAS: Architecture-Driven Weight Prediction for Neural Architecture Search
TLDR
This work proposes an Architecture-Driven Weight Prediction (ADWP) approach for neural architecture search and first design an architecture-intensive search space and then train a HyperNetwork by inputting stochastic encoding architecture parameters, which shows weights of convolution kernels can be well predicted for neural architectures in the search space. Expand
Task-Aware Performance Prediction for Efficient Architecture Search
TLDR
This work proposes a novel gradient-based framework for efficient architecture search by sharing information across several tasks by adopting a continuous parametrization of the model architecture, which allows for efficient gradient- based optimization. Expand
Sample-Efficient Neural Architecture Search by Learning Actions for Monte Carlo Tree Search.
TLDR
This paper proposes Latent Action Neural Architecture Search (LaNAS), which learns actions to recursively partition the search space into good or bad regions that contain networks with similar performance metrics. Expand
FENAS: Flexible and Expressive Neural Architecture Search
TLDR
This work proposes a novel architecture search algorithm called Flexible and Expressible Neural Architecture Search (FENAS), with more flexible and expressible search space than ENAS, in terms of more activation functions, input edges, and atomic operations. Expand
Recurrent Neural Architecture Search based on Randomness-Enhanced Tabu Algorithm
TLDR
This paper applies the randomness-enhanced tabu algorithm as a controller to sample candidate architectures, which balances the global exploration and local exploitation for the architectural solutions, and discovers the recurrent neural architecture within 0.78 GPU hour. Expand
Improving the Efficient Neural Architecture Search via Rewarding Modifications
TLDR
Improved-ENAS is proposed, a further improvement of ENAS that augments the reinforcement learning training method by modifying the reward of each tested architecture according to the results obtained in previously tested architectures. Expand
Efficient Neural Architecture Search with Network Morphism
TLDR
A novel framework enabling Bayesian optimization to guide the network morphism for efficient neural architecture search by introducing a neural network kernel and a tree-structured acquisition function optimization algorithm. Expand
Auto-Keras: An Efficient Neural Architecture Search System
TLDR
A novel framework enabling Bayesian optimization to guide the network morphism for efficient neural architecture search is proposed and an open-source AutoML system based on the developed framework is built, namely Auto-Keras. Expand
Graph HyperNetworks for Neural Architecture Search
TLDR
The GHN is proposed to amortize the search cost: given an architecture, it directly generates the weights by running inference on a graph neural network, which can predict network performance more accurately than regular hypernetworks and premature early stopping. Expand
Efficient Novelty-Driven Neural Architecture Search
TLDR
An efficient novelty search method for NAS is devised in this paper, and extensive experiments demonstrate the effectiveness and efficiency of the novelty search based architecture sampling method. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 53 REFERENCES
SMASH: One-Shot Model Architecture Search through HyperNetworks
TLDR
A technique to accelerate architecture selection by learning an auxiliary HyperNet that generates the weights of a main model conditioned on that model's architecture is proposed, achieving competitive performance with similarly-sized hand-designed networks. Expand
Efficient Architecture Search by Network Transformation
TLDR
This paper proposes a new framework toward efficient architecture search by exploring the architecture space based on the current network and reusing its weights, and employs a reinforcement learning agent as the meta-controller, whose action is to grow the network depth or layer width with function-preserving transformations. Expand
Neural Architecture Search with Reinforcement Learning
TLDR
This paper uses a recurrent network to generate the model descriptions of neural networks and trains this RNN with reinforcement learning to maximize the expected accuracy of the generated architectures on a validation set. Expand
Learning Time-Efficient Deep Architectures with Budgeted Super Networks
TLDR
This work proposes a new family of models called Budgeted Super Networks that are learned using reinforcement-learning inspired techniques applied to a budgeted learning objective function which includes the computation cost during disk/memory operations at inference. Expand
Accelerating Neural Architecture Search using Performance Prediction
TLDR
Standard frequentist regression models can predict the final performance of partially trained model configurations using features based on network architectures, hyperparameters, and time-series validation performance data and an early stopping method is proposed, which obtains a speedup of a factor up to 6x in both hyperparameter optimization and meta-modeling. Expand
Hierarchical Representations for Efficient Architecture Search
TLDR
This work efficiently discovers architectures that outperform a large number of manually designed models for image classification, obtaining top-1 error of 3.6% on CIFAR-10 and 20.3% when transferred to ImageNet, which is competitive with the best existing neural architecture search approaches. Expand
DeepArchitect: Automatically Designing and Training Deep Architectures
TLDR
This paper proposes an extensible and modular framework that allows the human expert to compactly represent complex search spaces over architectures and their hyperparameters and shows that the same search space achieves near state-of-the-art performance with a few samples. Expand
Designing Neural Network Architectures using Reinforcement Learning
TLDR
MetaQNN is introduced, a meta-modeling algorithm based on reinforcement learning to automatically generate high-performing CNN architectures for a given learning task that beat existing networks designed with the same layer types and are competitive against the state-of-the-art methods that use more complex layer types. Expand
Neural Optimizer Search with Reinforcement Learning
TLDR
An approach to automate the process of discovering optimization methods, with a focus on deep learning architectures, and introduces two new optimizers, named PowerSign and AddSign, which show transfer well and improve training on a variety of different tasks and architectures. Expand
Practical Network Blocks Design with Q-Learning
TLDR
This work provides a solution to automatically and efficiently design high performance network architectures by focusing on constructing network blocks, which can be stacked to generate the whole network. Expand
...
1
2
3
4
5
...