• Corpus ID: 49411844

DARTS: Differentiable Architecture Search

@article{Liu2018DARTSDA,
  title={DARTS: Differentiable Architecture Search},
  author={Hanxiao Liu and Karen Simonyan and Yiming Yang},
  journal={ArXiv},
  year={2018},
  volume={abs/1806.09055}
}
This paper addresses the scalability challenge of architecture search by formulating the task in a differentiable manner. Unlike conventional approaches of applying evolution or reinforcement learning over a discrete and non-differentiable search space, our method is based on the continuous relaxation of the architecture representation, allowing efficient search of the architecture using gradient descent. Extensive experiments on CIFAR-10, ImageNet, Penn Treebank and WikiText-2 show that our… 
[PDF] Semantic Reader
2,836 Citations
Highly Influential Citations
1,039
Background Citations
1,533
Methods Citations
1,693
Results Citations
105

Figures and Tables from this paper

2,836 Citations

DEff-ARTS: Differentiable Efficient ARchiTecture Search

Dff-ARTS is presented, a differentiable efficient architecture search method for automatically deriving CNN architectures for resource constrained devices that frame the search as a multi-objective optimisation problem where it minimise the classification loss and the computational complexity of performing inference on the target hardware.

RARTS: a Relaxed Architecture Search Method

This paper formulate a single level alternative and a relaxed architecture search (RARTS) method that utilizes training and validation datasets in architecture learning without involving mixed second derivatives of the corresponding loss functions.

DrNAS: Dirichlet Neural Architecture Search

This paper proposes a novel differentiable architecture search method by formulating it into a distribution learning problem, modeled by Dirichlet distribution, which improves the generalization ability and induces stochasticity that naturally encourages exploration in the search space.

Search Space Adaptation for Differentiable Neural Architecture Search in Image Classification

An adaptation scheme of the search space by introducing a search scope is proposed and the effectiveness of proposed method is demonstrated with ProxylessNAS for the image classification task and the trajectory of architecture parameter updates is visualize and provided to improve the architecture search.

Layered feature representation for differentiable architecture search

This work proposes an effective approach called Layered Feature Representation for Differentiable Architecture Search (LFR-DARTS), which iteratively search for multiple cell architectures from shallow to deep layers of the super network and can obtain the cell architecture at a specific network depth.

DATA: Differentiable ArchiTecture Approximation

Differentiable ArchiTecture Approximation (DATA) with an Ensemble Gumbel-Softmax (EGS) estimator is developed to automatically approximate architectures during searching and validating in a differentiable manner, guaranteeing the requisite efficiency during searching.

NASiam: Efficient Representation Learning using Neural Architecture Search for Siamese Networks

A novel approach that uses for the first time differentiable NAS to improve the multilayer perceptron projector and predictor architectures inside siamese-networks-based contrastive learning frameworks while preserving the simplicity of previous baselines is presented.

D-DARTS: Distributed Differentiable Architecture Search

This article proposes D-DARTS, a solution that addresses this problem by nesting neural networks at the cell level instead of using weight-sharing to produce more diversified and specialized architectures, and introduces a novel algorithm that can derive deeper architectures from a few trained cells, increasing performance and saving computation time.

DEGAS: differentiable efficient generator search

This work proposes an alternative strategy for GAN search by using a proxy task instead of common GAN training, called differentiable efficient generator search, which focuses on efficiently finding the generator in the GAN.

Differentiable Neural Architecture Search with Morphism-based Transformable Backbone Architectures

This study introduces a growing mechanism for differentiable neural architecture search based on network morphism that enables growing of the cell structures from small size towards large size ones with one-shot training and implements a recently proposed two-input backbone architecture for recurrent neural networks.
...

References

SHOWING 1-10 OF 48 REFERENCES

Progressive Neural Architecture Search

We propose a new method for learning the structure of convolutional neural networks (CNNs) that is more efficient than recent state-of-the-art methods based on reinforcement learning and evolutionary…

Efficient Architecture Search by Network Transformation

This paper proposes a new framework toward efficient architecture search by exploring the architecture space based on the current network and reusing its weights, and employs a reinforcement learning agent as the meta-controller, whose action is to grow the network depth or layer width with function-preserving transformations.

Understanding and Simplifying One-Shot Architecture Search

With careful experimental analysis, it is shown that it is possible to efficiently identify promising architectures from a complex search space without either hypernetworks or reinforcement learning controllers.

SMASH: One-Shot Model Architecture Search through HyperNetworks

A technique to accelerate architecture selection by learning an auxiliary HyperNet that generates the weights of a main model conditioned on that model's architecture is proposed, achieving competitive performance with similarly-sized hand-designed networks.

Neural Architecture Search with Reinforcement Learning

This paper uses a recurrent network to generate the model descriptions of neural networks and trains this RNN with reinforcement learning to maximize the expected accuracy of the generated architectures on a validation set.

Simple And Efficient Architecture Search for Convolutional Neural Networks

Surprisingly, this simple method to automatically search for well-performing CNN architectures based on a simple hill climbing procedure whose operators apply network morphisms, followed by short optimization runs by cosine annealing yields competitive results.

Hierarchical Representations for Efficient Architecture Search

This work efficiently discovers architectures that outperform a large number of manually designed models for image classification, obtaining top-1 error of 3.6% on CIFAR-10 and 20.3% when transferred to ImageNet, which is competitive with the best existing neural architecture search approaches.

Going deeper with convolutions

We propose a deep convolutional neural network architecture codenamed Inception that achieves the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition…

DeepArchitect: Automatically Designing and Training Deep Architectures

This paper proposes an extensible and modular framework that allows the human expert to compactly represent complex search spaces over architectures and their hyperparameters and shows that the same search space achieves near state-of-the-art performance with a few samples.

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

We propose an algorithm for meta-learning that is model-agnostic, in the sense that it is compatible with any model trained with gradient descent and applicable to a variety of different learning…