Meta-learning from Learning Curves Challenge: Lessons learned from the First Round and Design of the Second Round

  title={Meta-learning from Learning Curves Challenge: Lessons learned from the First Round and Design of the Second Round},
  author={Manh Hung Nguyen and Lisheng Sun and Nathan Grinsztajn and Isabelle Guyon},
Meta-learning from learning curves is an important yet often neglected research area in the Machine Learning community. We introduce a series of Reinforcement Learning-based meta-learning challenges, in which an agent searches for the best suited algorithm for a given dataset, based on feedback of learning curves from the environment. The first round attracted participants both from academia and industry. This paper an-alyzes the results of the first round (accepted to the competition program of… 



MetaREVEAL: RL-based Meta-learning from Learning Curves

This paper addresses a cornerstone of Automated Machine Learning: the problem of rapidly uncovering which machine learning algorithm performs best on a new dataset, and shows that the new RL-based meta-learning paradigm outperforms Free-Thaw and other baseline methods, with respect to the Area under the Learning curve metric.

Meta-Learning: A Survey

This chapter provides an overview of the state of the art in meta-learning, the science of systematically observing how different machine learning approaches perform on a wide range of learning tasks and then learning from this experience, or meta-data, to learn new tasks much faster than otherwise possible.

A Meta-Analysis of Overfitting in Machine Learning

This study conducts the first large meta-analysis of overfitting due to test set reuse in the machine learning community based on over one hundred machine learning competitions hosted on the Kaggle platform over the course of several years and shows little evidence of substantial overfitting.

Learning Curves for Decision Making in Supervised Machine Learning - A Survey

A framework that categorizes learning curve approaches using three criteria: the decision situation that they address, the intrinsic learning curve question that they answer and the type of resources that they use is contributed.

Lifelong Machine Learning

  • Zhiyuan ChenB. Liu
  • Computer Science, Education
    Synthesis Lectures on Artificial Intelligence and Machine Learning
  • 2016
As statistical machine learning matures, it is time to make a major effort to break the isolated learning tradition and to study lifelong learning to bring machine learning to new heights.

A Continual Learning Survey: Defying Forgetting in Classification Tasks

This work focuses on task incremental classification, where tasks arrive sequentially and are delineated by clear boundaries, and develops a novel framework to continually determine the stability-plasticity trade-off of the continual learner.

Generalizing from a Few Examples: A Survey on Few-Shot Learning

A thorough survey to fully understand Few-Shot Learning (FSL), and categorizes FSL methods from three perspectives: data, which uses prior knowledge to augment the supervised experience; model, which used to reduce the size of the hypothesis space; and algorithm, which using prior knowledgeto alter the search for the best hypothesis in the given hypothesis space.

Advances in MetaDL: AAAI 2021 challenge and workshop

The design of the challenge and its results are presented, and presentations made at the workshop are summarized, which focused on few-shot learning classification tasks of small images.

Winning Solutions and Post-Challenge Analyses of the ChaLearn AutoDL Challenge 2019

This paper reports the results and post-challenge analyses of ChaLearn’s AutoDL challenge series, which helped sorting out a profusion of AutoML solutions for Deep Learning (DL) that had been

Few-Shot Bayesian Optimization with Deep Kernel Surrogates

This work proposes the use of a deep kernel network for a Gaussian process surrogate that is meta-learned in an end-to-end fashion in order to jointly approximate the response functions of a collection of training data sets.