Corpus ID: 202660584

Continual learning: A comparative study on how to defy forgetting in classification tasks

@article{DeLange2019ContinualLA,
  title={Continual learning: A comparative study on how to defy forgetting in classification tasks},
  author={Matthias De Lange and Rahaf Aljundi and Marc Masana and Sarah Parisot and Xu Jia and Ale{\vs} Leonardis and Gregory G. Slabaugh and Tinne Tuytelaars},
  journal={ArXiv},
  year={2019},
  volume={abs/1909.08383}
}
Artificial neural networks thrive in solving the classification problem for a particular rigid task, where the network resembles a static entity of knowledge, acquired through generalized learning behaviour from a distinct training phase. [...] Key Method Our main contributions concern 1) a taxonomy and extensive overview of the state-of-the-art, 2) a novel framework to continually determine stability-plasticity trade-off of the continual learner, 3) a comprehensive experimental comparison of 10 state-of-the…Expand
Understanding the Role of Training Regimes in Continual Learning
TLDR
This work hypothesizes that the geometrical properties of the local minima found for each task play an important role in the overall degree of forgetting, and studies the effect of dropout, learning rate decay, and batch size, on forming training regimes that widen the tasks'Local minima and consequently, on helping it not to forget catastrophically. Expand
Prototypes-Guided Memory Replay for Continual Learning
  • Stella Ho, Ming Liu, Lan Du, Longxiang Gao, Yong Xiang
  • Computer Science
  • ArXiv
  • 2021
TLDR
A dynamic prototypesguided memory replay module is devised, incorporating it into an online meta-learning model, and the experimental results testify the superiority of the method in alleviating catastrophic forgetting and enabling efficient knowledge transfer. Expand
More Classifiers, Less Forgetting: A Generic Multi-classifier Paradigm for Incremental Learning
TLDR
A MUlti-Classifier (MUC) incremental learning paradigm that integrates an ensemble of auxiliary classifiers to estimate more effective regularization constraints, and extends two common methods, focusing on parameter and activation regularization, from the conventional single-classifier paradigm to MUC. Expand
GDumb: A Simple Approach that Questions Our Progress in Continual Learning
We discuss a general formulation for the Continual Learning (CL) problem for classification—a learning task where a stream provides samples to a learner and the goal of the learner, depending on theExpand
Meta-Consolidation for Continual Learning
TLDR
The authors' experiments with continual learning benchmarks of MNIST, CifAR-10, CIFAR-100 and Mini-ImageNet datasets show consistent improvement over five baselines, including a recent state-of-the-art, corroborating the promise of MERLIN. Expand
Remember More by Recalling Less: Investigating the Role of Batch Size in Continual Learning with Experience Replay (Student Abstract)
TLDR
It is found that small batches perform much better at stopping forgetting than larger batches, contrary to the intuitive assumption that it is better to recall more samples from the past to avoid forgetting. Expand
AlterSGD: Finding Flat Minima for Continual Learning by Alternative Training
TLDR
A simple yet effective optimization method, called AlterSGD, to search for a flat minima in the loss landscape, which can significantly mitigate the forgetting and outperform the state-of-the-art methods with a large margin under challenging continual learning protocols. Expand
Orthogonal Gradient Descent for Continual Learning
TLDR
The Orthogonal Gradient Descent (OGD) method is presented, which accomplishes this goal by projecting the gradients from new tasks onto a subspace in which the neural network output on previous task does not change and the projected gradient is still in a useful direction for learning the new task. Expand
Rectification-based Knowledge Retention for Continual Learning
TLDR
This work proposes a novel approach to address the task incremental learning problem, which involves training a model on new tasks that arrive in an incremental manner, and significantly outperforms the state-of-the-art task incremental generalized zero-shot learning method. Expand
Task-agnostic Continual Learning with Hybrid Probabilistic Models
TLDR
This work proposes HCL, a Hybrid generativediscriminative approach to Continual Learning for classification that uses the generative capabilities of the flow to avoid catastrophic forgetting through generative replay and a novel functional regularization technique. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 80 REFERENCES
Continual Learning with Tiny Episodic Memories
TLDR
It is observed that a very simple baseline, which jointly trains on both examples from the current task as well as examples stored in the memory, outperforms state-of-the-art CL approaches with and without episodic memory. Expand
Gradient Episodic Memory for Continual Learning
TLDR
A model for continual learning, called Gradient Episodic Memory (GEM) is proposed that alleviates forgetting, while allowing beneficial transfer of knowledge to previous tasks. Expand
Task-Free Continual Learning
TLDR
This work investigates how to transform continual learning to an online setup, and develops a system that keeps on learning over time in a streaming fashion, with data distributions gradually changing and without the notion of separate tasks. Expand
Overcoming catastrophic forgetting with hard attention to the task
TLDR
A task-based hard attention mechanism that preserves previous tasks' information without affecting the current task's learning, and features the possibility to control both the stability and compactness of the learned knowledge, which makes it also attractive for online learning or network compression applications. Expand
Reinforced Continual Learning
TLDR
A novel approach for continual learning is proposed, which searches for the best neural architecture for each coming task via sophisticatedly designed reinforcement learning strategies, and which outperforms existing continual learning alternatives for deep networks. Expand
Experience Replay for Continual Learning
TLDR
This work shows that using experience replay buffers for all past events with a mixture of on- and off-policy learning can still learn new tasks quickly yet can substantially reduce catastrophic forgetting in both Atari and DMLab domains, even matching the performance of methods that require task identities. Expand
An Empirical Investigation of Catastrophic Forgeting in Gradient-Based Neural Networks
TLDR
It is found that it is always best to train using the dropout algorithm--the drop out algorithm is consistently best at adapting to the new task, remembering the old task, and has the best tradeoff curve between these two extremes. Expand
Pseudo-Recursal: Solving the Catastrophic Forgetting Problem in Deep Neural Networks
TLDR
This work accomplishes pseudo-rehearsal by using a Generative Adversarial Network to generate items so that the deep network can learn to sequentially classify the CIFAR-10, SVHN and MNIST datasets. Expand
Online continual learning with no task boundaries
TLDR
This paper develops a solution to select a fixed number of constraints that they use to approximate the feasible region defined by the original constraints, and compares this approach against the methods that rely on task boundaries toSelect a fixed set of examples, and shows comparable or even better results. Expand
Continual Classification Learning Using Generative Models
TLDR
This work proposes a classification model that learns continuously from sequentially observed tasks, while preventing catastrophic forgetting, and builds on the lifelong generative capabilities of [10] and extends it to the classification setting by deriving a new variational bound on the joint log likelihood. Expand
...
1
2
3
4
5
...