Meta-Learning With Differentiable Convex Optimization

@article{Lee2019MetaLearningWD,
  title={Meta-Learning With Differentiable Convex Optimization},
  author={Kwonjoon Lee and Subhransu Maji and Avinash Ravichandran and Stefano Soatto},
  journal={2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2019},
  pages={10649-10657}
}
Many meta-learning approaches for few-shot learning rely on simple base learners such as nearest-neighbor classifiers. [] Key Method To efficiently solve the objective, we exploit two properties of linear classifiers: implicit differentiation of the optimality conditions of the convex problem and the dual formulation of the optimization problem. This allows us to use high-dimensional embeddings with improved generalization at a modest increase in computational overhead. Our approach, named MetaOptNet…

Figures and Tables from this paper

Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learning

TLDR
A simple process: meta-learning over a whole-classification pre-trained model on its evaluation metric achieves competitive performance to state-of-the-art methods on standard bench-marks and sheds some light on understanding the trade-offs between the meta- learning objective and the whole- classification objective in few-shot learning.

Transductive Few-Shot Learning: Clustering is All You Need?

TLDR
A general formulation for clustering and transductive few-shot learning, which integrates prototype-based objectives, Laplacian regularization and supervision constraints from a few labeled data points, and derives a computationally efficient block-coordinate bound optimizer, with convergence guarantee.

Revisiting Deep Local Descriptor for Improved Few-Shot Classification

TLDR
It is shown that the reliance on sophisticated classifier is not necessary and a simple classifier applied directly to improved feature embeddings can outperform state-of-the-art methods.

Meta Generalized Network for Few-Shot Classification

TLDR
This paper develops a meta backbone training method that learns a flexible feature extractor and a classifier initializer efficiently, delightedly leading to fast adaption to unseen few-shot tasks without overfitting, and designs a trainable adaptive interval model to improve the cosine classifier, which increases the recognition accuracy of hard examples.

Shallow Bayesian Meta Learning for Real-World Few-Shot Recognition

TLDR
MetaQDA is introduced, a Bayesian meta-learning generalisation of the classic quadratic discriminant analysis that leads to excellent performance in cross-domain few-shot learning, class-incremental few- shot learning, and crucially for real-world applications, the Bayesian formulation leads to state-of-the-art uncertainty calibration in predictions.

Revisiting Unsupervised Meta-Learning: Amplifying or Compensating for the Characteristics of Few-Shot Tasks

TLDR
This work finds that the base class set labels are not necessary, and discriminative embeddings could be meta-learned in an unsupervised manner, and two modifications -- the semi-normalized distance metric and the sufficient sampling -- improves un supervised meta-learning (UML) significantly.

Class-Discriminative Feature Embedding For Meta-Learning based Few-Shot Classification

  • Alireza RahimpourH. Qi
  • Computer Science
    2020 IEEE Winter Conference on Applications of Computer Vision (WACV)
  • 2020
TLDR
A few-shot learning framework based on structured margin loss which takes into account the global structure of the support set in order to generate a highly discriminative feature space where the features from distinct classes are well separated in clusters is proposed.

Fast Few-Shot Classification by Few-Iteration Meta-Learning

TLDR
This work is the first to integrate both induction and transduction into the base learner in an optimization-based meta-learning framework, and demonstrates the speed and effectiveness of this approach on four few-shot classification datasets.

Meta-free few-shot learning via representation learning with weight averaging

TLDR
A new transfer learning method to obtain accurate and reliable models for few-shot regression and classification is proposed, called meta-free representation learning (MFRL), which achieves the highest accuracy on a wide range of few- shot learning benchmark datasets.

Beyond Simple Meta-Learning: Multi-Purpose Models for Multi-Domain, Active and Continual Few-Shot Learning

TLDR
This work proposes a variance-sensitive class of models that operate in a low-label regime and employs a hierarchically regularized Mahalanobis-distance based classifier combined with a state of the art neural adaptive feature extractor to achieve strong performance on Meta-Dataset, mini-Image net and tiered-ImageNet benchmarks.
...

References

SHOWING 1-10 OF 35 REFERENCES

Optimization as a Model for Few-Shot Learning

Learning to Compare: Relation Network for Few-Shot Learning

TLDR
A conceptually simple, flexible, and general framework for few-shot learning, where a classifier must learn to recognise new classes given only few examples from each, which is easily extended to zero- shot learning.

Meta-learning with differentiable closed-form solvers

TLDR
The main idea is to teach a deep network to use standard machine learning tools, such as ridge regression, as part of its own internal model, enabling it to quickly adapt to novel data.

Meta-Learning for Semi-Supervised Few-Shot Classification

TLDR
This work proposes novel extensions of Prototypical Networks that are augmented with the ability to use unlabeled examples when producing prototypes, and confirms that these models can learn to improve their predictions due to unlabeling examples, much like a semi-supervised algorithm would.

Transductive Propagation Network for Few-shot Learning

TLDR
This paper proposes Transductive Propagation Network (TPN), a transductive method that classifies the entire test set at once to alleviate the low-data problem and explicitly learns an underlying manifold space that is appropriate to propagate labels from few-shot examples.

TADAM: Task dependent adaptive metric for improved few-shot learning

TLDR
This work identifies that metric scaling and metric task conditioning are important to improve the performance of few-shot algorithms and proposes and empirically test a practical end-to-end optimization procedure based on auxiliary task co-training to learn a task-dependent metric space.

Prototypical Networks for Few-shot Learning

TLDR
This work proposes Prototypical Networks for few-shot classification, and provides an analysis showing that some simple design decisions can yield substantial improvements over recent approaches involving complicated architectural choices and meta-learning.

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

We propose an algorithm for meta-learning that is model-agnostic, in the sense that it is compatible with any model trained with gradient descent and applicable to a variety of different learning

Matching Networks for One Shot Learning

TLDR
This work employs ideas from metric learning based on deep neural features and from recent advances that augment neural networks with external memories to learn a network that maps a small labelled support set and an unlabelled example to its label, obviating the need for fine-tuning to adapt to new class types.

A Simple Neural Attentive Meta-Learner

TLDR
This work proposes a class of simple and generic meta-learner architectures that use a novel combination of temporal convolutions and soft attention; the former to aggregate information from past experience and the latter to pinpoint specific pieces of information.