Teacher Guided Architecture Search

@article{Bashivan2019TeacherGA,
  title={Teacher Guided Architecture Search},
  author={Pouya Bashivan and Mark Tensen and James J. DiCarlo},
  journal={2019 IEEE/CVF International Conference on Computer Vision (ICCV)},
  year={2019},
  pages={5319-5328}
}
Much of the recent improvement in neural networks for computer vision has resulted from discovery of new networks architectures. Most prior work has used the performance of candidate models following limited training to automatically guide the search in a feasible way. Could further gains in computational efficiency be achieved by guiding the search via measurements of a high performing network with unknown detailed architecture (e.g. the primate visual system)? As one step toward this goal, we… Expand
Guided Evolution for Neural Architecture Search
TLDR
G-EA forces exploitation of the most performant networks by descendant generation while at the same time forcing exploration by parent mutation and by favouring younger architectures to the detriment of older ones, allowing the search to be efficiently guided. Expand
Teacher Guided Neural Architecture Search for Face Recognition
TLDR
This paper develops a novel teacher guided neural architecture search method to directly search the student network with flexible channel and layer sizes and defines the search space as the number of channels/layers, which is sampled based on the probability distribution and is learned by minimizing the search objective of theStudent network. Expand
A Comprehensive Survey of Neural Architecture Search
  • Pengzhen Ren, Yun Xiao, +4 authors Xin Wang
  • Computer Science
  • ACM Comput. Surv.
  • 2021
TLDR
This survey provides a new perspective on Neural Architecture Search, beginning with an overview of the characteristics of the earliest NAS algorithms, summarizing the problems in these earlyNAS algorithms, and then providing solutions for subsequent related research work. Expand
CORnet: Modeling the Neural Mechanisms of Core Object Recognition
TLDR
The current best ANN model derived from this approach (CORnet-S) is among the top models on Brain-Score, a composite benchmark for comparing models to the brain, but is simpler than other deep ANNs in terms of the number of convolutions performed along the longest path of information processing in the model. Expand
Knowledge Distillation: A Survey
TLDR
A comprehensive survey of knowledge distillation from the perspectives of knowledge categories, training schemes, distillation algorithms and applications is provided. Expand
Shapeshifter Networks: Cross-layer Parameter Sharing for Scalable and Effective Deep Learning
TLDR
SSNs address the observation that many neural networks are severely overparameterized, resulting in significant waste in computational resources as well as being susceptible to overfitting by learning where and how to share parameters between layers in a neural network while avoiding degenerate solutions that result in underfitting. Expand
Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
TLDR
A literature review on the application of NAS to computer vision problems is provided and existing approaches are summarized into several categories according to their efforts in bridging the gap. Expand
Shapeshifter Networks: Decoupling Layers from Parameters for Scalable and Effective Deep Learning.
TLDR
This work presents Shapeshifter Networks (SSNs), a flexible neural network framework that decouples layers from model weights, enabling it to implement any neural network with an arbitrary number of parameters. Expand
A Neurobiological Evaluation Metric for Neural Network Model Search
TLDR
A human-model similarity (HMS) metric is introduced, which quantifies the similarity of human fMRI and network activation behavior, and is shown to be correlated with better performance on two computer vision tasks: next frame prediction and object matching accuracy. Expand
Reconsidering CO2 emissions from Computer Vision
TLDR
This work analyzes the total cost of CO2 emissions and proposes adding “enforcement” as a pillar of ethical AI and provides some recommendations for how architecture designers and broader CV community can curb the climate crisis. Expand
...
1
2
...

References

SHOWING 1-10 OF 58 REFERENCES
Progressive Neural Architecture Search
We propose a new method for learning the structure of convolutional neural networks (CNNs) that is more efficient than recent state-of-the-art methods based on reinforcement learning and evolutionaryExpand
Neural Architecture Search with Reinforcement Learning
TLDR
This paper uses a recurrent network to generate the model descriptions of neural networks and trains this RNN with reinforcement learning to maximize the expected accuracy of the generated architectures on a validation set. Expand
Regularized Evolution for Image Classifier Architecture Search
TLDR
This work evolves an image classifier---AmoebaNet-A---that surpasses hand-designs for the first time and gives evidence that evolution can obtain results faster with the same hardware, especially at the earlier stages of the search. Expand
Rethinking the Inception Architecture for Computer Vision
TLDR
This work is exploring ways to scale up networks in ways that aim at utilizing the added computation as efficiently as possible by suitably factorized convolutions and aggressive regularization. Expand
Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition
TLDR
These evaluations show that, unlike previous bio-inspired models, the latest DNNs rival the representational performance of IT cortex on this visual object recognition task and propose an extension of “kernel analysis” that measures the generalization accuracy as a function of representational complexity. Expand
IRLAS: Inverse Reinforcement Learning for Architecture Search
TLDR
An inverse reinforcement learning method for architecture search (IRLAS), which trains an agent to learn to search network structures that are topologically inspired by human-designed network to extract the abstract topological knowledge of an expert human-design network (ResNeXt). Expand
DARTS: Differentiable Architecture Search
TLDR
The proposed algorithm excels in discovering high-performance convolutional architectures for image classification and recurrent architectures for language modeling, while being orders of magnitude faster than state-of-the-art non-differentiable techniques. Expand
Efficient Neural Architecture Search via Parameter Sharing
TLDR
Efficient Neural Architecture Search is a fast and inexpensive approach for automatic model design that establishes a new state-of-the-art among all methods without post-training processing and delivers strong empirical performances using much fewer GPU-hours. Expand
SMASH: One-Shot Model Architecture Search through HyperNetworks
TLDR
A technique to accelerate architecture selection by learning an auxiliary HyperNet that generates the weights of a main model conditioned on that model's architecture is proposed, achieving competitive performance with similarly-sized hand-designed networks. Expand
Large-Scale Evolution of Image Classifiers
TLDR
It is shown that it is now possible to evolve models with accuracies within the range of those published in the last year, starting from trivial initial conditions and reaching accuracies of 94.6% and 77.0%, respectively. Expand
...
1
2
3
4
5
...