Unchain the Search Space with Hierarchical Differentiable Architecture Search

@article{Liu2021UnchainTS,
  title={Unchain the Search Space with Hierarchical Differentiable Architecture Search},
  author={Guanting Liu and Yujie Zhong and Sheng Guo and Matthew R. Scott and Weilin Huang},
  journal={ArXiv},
  year={2021},
  volume={abs/2101.04028}
}
Differentiable architecture search (DAS) has made great progress in searching for high-performance architectures with reduced computational cost. However, DAS-based methods mainly focus on searching for a repeatable cell structure, which is then stacked sequentially in multiple stages to form the networks. This configuration significantly reduces the search space, and ignores the importance of connections between the cells. To overcome this limitation, in this paper, we propose a Hierarchical… 

Mutually-aware Sub-Graphs Differentiable Architecture Search

This paper proposes a conceptually simple yet efficient method to bridge these two paradigms, referred as Mutually-aware Sub-Graphs Differentiable Architecture Search (MSG-DAS), and introduces a memoryefficient super-net guidance distillation to improve training.

References

SHOWING 1-10 OF 43 REFERENCES

Progressive Differentiable Architecture Search: Bridging the Depth Gap Between Search and Evaluation

This paper presents an efficient algorithm which allows the depth of searched architectures to grow gradually during the training procedure, and solves two issues, namely, heavier computational overheads and weaker search stability, which are solved using search space approximation and regularization.

PC-DARTS: Partial Channel Connections for Memory-Efficient Differentiable Architecture Search

This paper presents a novel approach, namely, Partially-Connected DARTS, by sampling a small part of super-network to reduce the redundancy in exploring the network space, thereby performing a more efficient search without comprising the performance.

SNAS: Stochastic Neural Architecture Search

It is proved that this search gradient optimizes the same objective as reinforcement-learning-based NAS, but assigns credits to structural decisions more efficiently, and is further augmented with locally decomposable reward to enforce a resource-efficient constraint.

ISTA-NAS: Efficient and Consistent Neural Architecture Search by Sparse Coding

This paper performs the differentiable search on a compressed lower-dimensional space that has the same validation loss as the original sparse solution space, and recover an architecture by solving the sparse coding problem in an alternate manner.

Understanding and Robustifying Differentiable Architecture Search

It is shown that by adding one of various types of regularization to DARTS, one can robustify DARTS to find solutions with less curvature and better generalization properties, and proposes several simple variations of DARTS that perform substantially more robustly in practice.

DARTS+: Improved Differentiable Architecture Search with Early Stopping

It is claimed that there exists overfitting in the optimization of DARTS, and a simple and effective algorithm is proposed, named "DARTS+", to avoid the collapse and improve the original DARts, by "early stopping" the search procedure when meeting a certain criterion.

Searching for a Robust Neural Architecture in Four GPU Hours

  • Xuanyi DongYezhou Yang
  • Computer Science
    2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2019
The approach can be trained in an end-to-end fashion by gradient descent, named Gradient-based search using Differentiable Architecture Sampler (GDAS), and the discovered model obtains a test error of 2.82% with only 2.5M parameters, which is on par with the state-of-the-art.

Hierarchical Representations for Efficient Architecture Search

This work efficiently discovers architectures that outperform a large number of manually designed models for image classification, obtaining top-1 error of 3.6% on CIFAR-10 and 20.3% when transferred to ImageNet, which is competitive with the best existing neural architecture search approaches.

DARTS: Differentiable Architecture Search

The proposed algorithm excels in discovering high-performance convolutional architectures for image classification and recurrent architectures for language modeling, while being orders of magnitude faster than state-of-the-art non-differentiable techniques.

Efficient Neural Architecture Search via Parameter Sharing

Efficient Neural Architecture Search is a fast and inexpensive approach for automatic model design that establishes a new state-of-the-art among all methods without post-training processing and delivers strong empirical performances using much fewer GPU-hours.