Self-Supervised Representation Learning for Evolutionary Neural Architecture Search

@article{Wei2021SelfSupervisedRL,
  title={Self-Supervised Representation Learning for Evolutionary Neural Architecture Search},
  author={Chen Wei and Yiping Tang and Chuang Niu and Haihong Hu and Yue Wang and Jimin Liang},
  journal={IEEE Computational Intelligence Magazine},
  year={2021},
  volume={16},
  pages={33-49}
}
Recently proposed neural architecture search (NAS) algorithms adopt neural predictors to accelerate architecture search. The capability of neural predictors to accurately predict the performance metrics of the neural architecture is critical to NAS, but obtaining training datasets for neural predictors is often time-consuming. How to obtain a neural predictor with high prediction accuracy using a small amount of training data is a central problem to neural predictor-based NAS. Here, a new… 

A Survey on Evolutionary Construction of Deep Neural Networks

An insight is provided into the automated DNN construction process by formulating it into a multi-level multi-objective large-scale optimization problem with constraints, where the non-convex, non-differentiable and black-box nature of this problem makes evolutionary algorithms (EAs) to stand out as a promising solver.

BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search

This work presents Block-wisely Self-supervised Neural Architecture Search (BossNAS), an unsupervised NAS method that addresses the problem of in-accurate architecture rating caused by large weight-sharing space and biased supervision in previous methods

Contrastive Embeddings for Neural Architectures

This work uses contrastive learning to identify networks across different initializations based on their data Jacobians, and automatically produces the first architecture embeddings independent from the parametrization of the search space.

CATE: Computation-aware Neural Architecture Encoding with Transformers

This work introduces a Computation-Aware Transformer-based Encoding method called CATE, which employs a pairwise pre-training scheme to learn computation-aware encodings using Transformers with cross-attention that is beneficial to the downstream search, especially in the large search space.

Anomaly detection in IR images of PV modules using supervised contrastive learning

This work serves the community with a more realistic view on PV module fault detection using unsupervised domain adaptation to develop more performant methods with favorable generalization capabilities.

Evolutionary deep learning: A survey

References

SHOWING 1-10 OF 49 REFERENCES

NPENAS: Neural Predictor Guided Evolution for Neural Architecture Search

This article proposes a neural predictor guided EA to enhance the exploration ability of EA for NAS (NPENAS) and designs two kinds of neural predictors, including a graph-based uncertainty estimation network as the surrogate model and introduces a new random architecture sampling method to overcome the drawbacks of the existing sampling method.

Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?

This work finds empirically that pre-training architecture representations using only neural architectures without their accuracies as labels considerably improve the downstream architecture search efficiency and visualize how unsupervised architecture representation learning better encourages neural architectures with similar connections and operators to cluster together.

Evolving Deep Convolutional Neural Networks for Image Classification

A new method using genetic algorithms for evolving the architectures and connection weight initialization values of a deep convolutional neural network to address image classification problems and a novel fitness evaluation method is proposed to speed up the heuristic search with substantially less computational resource.

Efficient Neural Architecture Search via Parameter Sharing

Efficient Neural Architecture Search is a fast and inexpensive approach for automatic model design that establishes a new state-of-the-art among all methods without post-training processing and delivers strong empirical performances using much fewer GPU-hours.

Neural Predictor for Neural Architecture Search

This work proposes an approach with three basic steps that is more than 20 times as sample efficient as Regularized Evolution on the NASBench-101 benchmark and can compete on ImageNet with more complex approaches based on weight sharing, such as ProxylessNAS.

Accelerating Neural Architecture Search using Performance Prediction

Standard frequentist regression models can predict the final performance of partially trained model configurations using features based on network architectures, hyperparameters, and time-series validation performance data and an early stopping method is proposed, which obtains a speedup of a factor up to 6x in both hyperparameter optimization and meta-modeling.

Neural Architecture Search with Reinforcement Learning

This paper uses a recurrent network to generate the model descriptions of neural networks and trains this RNN with reinforcement learning to maximize the expected accuracy of the generated architectures on a validation set.

Neural Architecture Search Using Deep Neural Networks and Monte Carlo Tree Search

A novel scalable Monte Carlo Tree Search (MCTS) based NAS agent, named AlphaX, to tackle the search efficiency and the network evaluation cost, and improves a variety of vision applications from Neural Style Transfer, to Image Captioning and Object Detection.

BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search

A thorough analysis of the "BO + neural predictor framework" is given by identifying five main components: the architecture encoding, neural predictor, uncertainty calibration method, acquisition function, and acquisition function optimization, and developing a novel path-based encoding scheme for neural architectures.

Progressive Neural Architecture Search

We propose a new method for learning the structure of convolutional neural networks (CNNs) that is more efficient than recent state-of-the-art methods based on reinforcement learning and evolutionary