Neural Architecture Transfer

@article{Lu2021NeuralAT,
  title={Neural Architecture Transfer},
  author={Zhichao Lu and Gautam Sreekumar and Erik D. Goodman and Wolfgang Banzhaf and Kalyanmoy Deb and Vishnu Naresh Boddeti},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year={2021},
  volume={43},
  pages={2971-2989}
}
Neural architecture search (NAS) has emerged as a promising avenue for automatically designing task-specific neural networks. Existing NAS approaches require one complete search for each deployment specification of hardware or objective. This is a computationally impractical endeavor given the potentially large number of application scenarios. In this paper, we propose <italic>Neural Architecture Transfer</italic> (NAT) to overcome this limitation. NAT is designed to efficiently generate task… 
Neural Network Design: Learning from Neural Architecture Search
TLDR
This paper argues for the necessity of studying the landscape property of NAS and proposes to use the so-called Exploratory Landscape Analysis (ELA) techniques for this goal, taking a broad set of designs of the deep convolutional network.
A Comprehensive Survey on Hardware-Aware Neural Architecture Search
TLDR
This survey provides a detailed review of existing HW-NAS research and categorizes them according to four key dimensions: the search space, the search strategy, the acceleration technique, and the hardware cost estimation strategies.
Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
TLDR
A literature review on the application of NAS to computer vision problems is provided and existing approaches are summarized into several categories according to their efforts in bridging the gap.
Pareto-Frontier-aware Neural Architecture Generation for Diverse Budgets
TLDR
A Pareto-Frontier-aware Neural Architecture Generator (NAG) is proposed which takes an arbitrary budget as input and produces the Pare to optimal architecture for the target budget.
Large Scale Neural Architecture Search with Polyharmonic Splines
TLDR
This work proposes a NAS method based on polyharmonic splines that can perform search directly on large scale, imbalanced target datasets, and designed a model which achieved a top-1 accuracy of 40.03% on ImageNet22K.
FUSIONET: A Scalable Framework for Image Classification
TLDR
This paper has proposed a unique framework, FUSIONET, to aid in image classification, trained, tested, and evaluated on real-world datasets, achieving state-of-the-art on the popular CINIC-10 dataset.
Attentional Feature Fusion
TLDR
This work proposes a multiscale channel attention module, which addresses issues that arise when fusing features given at different scales, and demonstrates that the initial integration of feature maps can become a bottleneck and that this issue can be alleviated by adding another level of attention.
SpinalNet: Deep Neural Network with Gradual Input
TLDR
The human somatosensory system is studied and the SpinalNet is proposed to achieve higher accuracy with less computational resources and the vanishing gradient problem does not exist.
Reconsidering CO2 emissions from Computer Vision
TLDR
This work analyzes the total cost of CO2 emissions and proposes adding “enforcement” as a pillar of ethical AI and provides some recommendations for how architecture designers and broader CV community can curb the climate crisis.
Accelerating Multi-Objective Neural Architecture Search by Random-Weight Evaluation
  • Shengran Hu, Ran Cheng, Cheng He, Zhichao Lu, Jing Wang, Miao Zhang
  • Computer Science
    Complex & Intelligent Systems
  • 2021
TLDR
A new performance estimation metric, named Random-Weight Evaluation (RWE) is introduced to quantify the quality of CNNs in a cost-efficient manner and reveals the effectiveness of the proposed RWE in estimating the performance compared with existing methods.
...
1
2
3
...

References

SHOWING 1-10 OF 93 REFERENCES
Towards Fast Adaptation of Neural Architectures with Meta Learning
TLDR
A novel Transferable Neural Architecture Search method based on meta-learning, which learns a meta-architecture that is able to adapt to a new task quickly through a few gradient steps, which makes the transferred architecture suitable for the specific task.
Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution
TLDR
This work proposes LEMONADE, an evolutionary algorithm for multi-objective architecture search that allows approximating the entire Pareto-front of architectures under multiple objectives, such as predictive performance and number of parameters, in a single run of the method.
BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models
TLDR
The proposed BigNAS, an approach that challenges the conventional wisdom that post-processing of the weights is necessary to get good prediction accuracies, is proposed, able to train a single set of shared weights on ImageNet and use these weights to obtain child models whose sizes range from 200 to 1000 MFLOPs.
XferNAS: Transfer Neural Architecture Search
TLDR
This work proposes a generally applicable framework that introduces only minor changes to existing optimizers to leverage this feature of Neural Architecture Search and demonstrates that the proposed framework generally gives better results and, in the worst case, is just as good as the unmodified optimizer.
ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
TLDR
ProxylessNAS is presented, which can directly learn the architectures for large-scale target tasks and target hardware platforms and apply ProxylessNAS to specialize neural architectures for hardware with direct hardware metrics (e.g. latency) and provide insights for efficient CNN architecture design.
Fast Task-Aware Architecture Inference
TLDR
This work proposes a novel gradient-based framework for efficient architecture search by sharing information across several tasks by adopting a continuous parametrization of the model architecture which allows for efficient gradient- based optimization.
EAT-NAS: Elastic Architecture Transfer for Accelerating Large-scale Neural Architecture Search
TLDR
This work proposes an elastic architecture transfer mechanism for accelerating large-scale neural architecture search (EAT-NAS), which takes only less than 5 days on 8 TITAN X GPUs, which is significantly less than the computational consumption of the state-of-the-art large- scale NAS methods.
Efficient Neural Architecture Search via Parameter Sharing
TLDR
Efficient Neural Architecture Search is a fast and inexpensive approach for automatic model design that establishes a new state-of-the-art among all methods without post-training processing and delivers strong empirical performances using much fewer GPU-hours.
Block-Wisely Supervised Neural Architecture Search With Knowledge Distillation
TLDR
This work proposes to modularize the large search space of NAS into blocks to ensure that the potential candidate architectures are fully trained, and distill the neural architecture (DNA) knowledge from a teacher model to supervise the block-wise architecture search, which significantly improves the effectiveness of NAS.
Evaluating the Search Phase of Neural Architecture Search
TLDR
This paper finds that on average, the state-of-the-art NAS algorithms perform similarly to the random policy; the widely-used weight sharing strategy degrades the ranking of the NAS candidates to the point of not reflecting their true performance, thus reducing the effectiveness of the search process.
...
1
2
3
4
5
...