Meta-Learning of Neural Architectures for Few-Shot Learning

@article{Elsken2020MetaLearningON,
  title={Meta-Learning of Neural Architectures for Few-Shot Learning},
  author={T. Elsken and B. Staffler and J. H. Metzen and F. Hutter},
  journal={2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2020},
  pages={12362-12372}
}
  • T. Elsken, B. Staffler, +1 author F. Hutter
  • Published 2020
  • Computer Science, Mathematics
  • 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
The recent progress in neural architecture search (NAS) has allowed scaling the automated design of neural architectures to real-world domains, such as object detection and semantic segmentation. However, one prerequisite for the application of NAS are large amounts of labeled data and compute resources. This renders its application challenging in few-shot learning scenarios, where many related tasks need to be learned, each with limited amounts of data and compute time. Thus, few-shot learning… Expand

Figures and Tables from this paper

Rapid Neural Architecture Search by Learning to Generate Graphs from Datasets
TLDR
The proposed MetaD2A (Meta Dataset-to-Architecture) model can stochastically generate graphs from a given set (dataset) via a cross-modal latent space learned with amortized meta-learning and also proposes a meta-performance predictor to estimate and select the best architecture without direct training on target datasets. Expand
A survey on data‐efficient algorithms in big data era
TLDR
This work investigates the issue of algorithms’ data hungriness, presents a comprehensive review of existing data-efficient methods and systematizes them into four categories, and delineates the limitations, discusses research challenges, and suggests future opportunities to advance the research on data-efficiency in machine learning. Expand
Bag of Tricks for Neural Architecture Search
TLDR
Practical considerations that help improve the stability, efficiency and overall performance of neural architecture search methods are discussed. Expand
Differentiable Neural Architecture Search for High-Dimensional Time Series Forecasting
TLDR
A novel two-input backbone cell architecture for recurrent neural networks is proposed, in which each candidate operation is also carried out with two inputs, and is shown to be more effective compared to the classic differentiable architecture search (DARTS) in both uni-variate and multi-Variate time series prediction tasks. Expand
Differentiable Neural Architecture Search with Morphism-based Transformable Backbone Architectures
TLDR
This study introduces a growing mechanism for differentiable neural architecture search based on network morphism that enables growing of the cell structures from small size towards large size ones with one-shot training and implements a recently proposed two-input backbone architecture for recurrent neural networks. Expand
ECKPN: Explicit Class Knowledge Propagation Network for Transductive Few-shot Learning
TLDR
An Explicit Class Knowledge Propagation Network (ECKPN), which is composed of the comparison, squeeze and calibration modules, to address the problem of exploring the classlevel knowledge that can be easily learned by humans from just a handful of samples. Expand
Ensemble Making Few-Shot Learning Stronger
TLDR
An ensemble approach to reduce the variance and fine-tuning and feature attention strategies to calibrate relationlevel features are explored and shown that this model significantly outperforms the previous state-of-theart models. Expand
External-Memory Networks for Low-Shot Learning of Targets in Forward-Looking-Sonar Imagery
TLDR
A memory-based framework for real-time, data-efficient target analysis in forward-looking-sonar (FLS) imagery relies on first removing non-discriminative details from the imagery using a small-scale DENSENET-inspired network, which cascades into a novel NEURALRAM-based convolutional matching network, NRMN, for low-shot target recognition. Expand
Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharing
TLDR
A new method is introduced, FedEx, to accelerate federated hyperparameter tuning that is applicable to widely-used federated optimization methods such as FedAvg and recent variants and correctly tunes the on-device learning rate in the setting of online convex optimization across devices. Expand
...
1
2
3
...

References

SHOWING 1-10 OF 70 REFERENCES
DARTS: Differentiable Architecture Search
TLDR
The proposed algorithm excels in discovering high-performance convolutional architectures for image classification and recurrent architectures for language modeling, while being orders of magnitude faster than state-of-the-art non-differentiable techniques. Expand
Auto-Meta: Automated Gradient Based Meta Learner Search
TLDR
This work verifies that automated architecture search synergizes with the effect of gradient-based meta learning, and adopts the progressive neural architecture search to find optimal architectures for meta-learners. Expand
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
We propose an algorithm for meta-learning that is model-agnostic, in the sense that it is compatible with any model trained with gradient descent and applicable to a variety of different learningExpand
Towards Fast Adaptation of Neural Architectures with Meta Learning
TLDR
A novel Transferable Neural Architecture Search method based on meta-learning, which learns a meta-architecture that is able to adapt to a new task quickly through a few gradient steps, which makes the transferred architecture suitable for the specific task. Expand
Aging Evolution for Image Classifier Architecture Search
How to train your MAML
TLDR
This paper proposes various modifications to MAML that not only stabilize the system, but also substantially improve the generalization performance, convergence speed and computational overhead of MAMl, which it is called M AML++. Expand
ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
TLDR
ProxylessNAS is presented, which can directly learn the architectures for large-scale target tasks and target hardware platforms and apply ProxylessNAS to specialize neural architectures for hardware with direct hardware metrics (e.g. latency) and provide insights for efficient CNN architecture design. Expand
SNAS: Stochastic Neural Architecture Search
TLDR
It is proved that this search gradient optimizes the same objective as reinforcement-learning-based NAS, but assigns credits to structural decisions more efficiently, and is further augmented with locally decomposable reward to enforce a resource-efficient constraint. Expand
On First-Order Meta-Learning Algorithms
TLDR
A family of algorithms for learning a parameter initialization that can be fine-tuned quickly on a new task, using only first-order derivatives for the meta-learning updates, including Reptile, which works by repeatedly sampling a task, training on it, and moving the initialization towards the trained weights on that task. Expand
Meta-Learning in Neural Networks: A Survey
TLDR
A new taxonomy is proposed that provides a more comprehensive breakdown of the space of meta-learning methods today, including few-shot learning, reinforcement learning and architecture search, and promising applications and successes. Expand
...
1
2
3
4
5
...