Continuous Learning of Context-dependent Processing in Neural Networks

@article{Zeng2019ContinuousLO,
  title={Continuous Learning of Context-dependent Processing in Neural Networks},
  author={Guanxiong Zeng and Yang Chen and Bo Cui and Shan Yu},
  journal={ArXiv},
  year={2019},
  volume={abs/1810.01256}
}
Deep artificial neural networks (DNNs) are powerful tools for recognition and classification as they learn sophisticated mapping rules between the inputs and the outputs. [...] Key Result This would enable highly compact systems to gradually learn myriad of regularities of the real world and eventually behave appropriately within it.Expand
A Unified Framework for Lifelong Learning in Deep Neural Networks
TLDR
This paper proposes a simple yet powerful unified framework that demonstrates all of these desirable properties of lifelong learning, including non-forgetting, concept rehearsal, forward transfer and backward transfer of knowledge, and so on. Expand
Gradient Projection Memory for Continual Learning
TLDR
This work proposes a novel approach where a neural network learns new tasks by taking gradient steps in the orthogonal direction to the gradient subspaces deemed important for the past tasks, and shows that this induces minimum to no interference with thepast tasks, thereby mitigates forgetting. Expand
Orthogonal Gradient Descent for Continual Learning
TLDR
The Orthogonal Gradient Descent (OGD) method is presented, which accomplishes this goal by projecting the gradients from new tasks onto a subspace in which the neural network output on previous task does not change and the projected gradient is still in a useful direction for learning the new task. Expand
Efficient and robust multi-task learning in the brain with modular task primitives
TLDR
This work shows that a modular network endowed with task primitives allows for learning multiple tasks well while keeping parameter counts, and updates, low, and makes predictions for novel neuroscience experiments in which targeted perturbations are employed to explore solution spaces. Expand
LEARNING WITH LONG-TERM REMEMBERING: FOL-
Current deep neural networks can achieve remarkable performance on a single task. However, when the deep neural network is continually trained on a sequence of tasks, it seems to gradually forget theExpand
Continual Learning Using Task Conditional Neural Networks
TLDR
This work proposes Task Conditional Neural Networks (TCNN) that does not require to known the reoccurring tasks in advance and outperforms the state-of-the-art solutions in continual learning and adapting to new tasks that are not defined in advance. Expand
Learning with Long-term Remembering: Following the Lead of Mixed Stochastic Gradient
TLDR
A novel and effective lifelong learning algorithm, calledMixEd stochastic GrAdient (MEGA), which allows deep neural networks to ac-quire the ability of retaining performance on old tasks while learning new tasks. Expand
Understanding the Role of Training Regimes in Continual Learning
TLDR
This work hypothesizes that the geometrical properties of the local minima found for each task play an important role in the overall degree of forgetting, and studies the effect of dropout, learning rate decay, and batch size, on forming training regimes that widen the tasks'Local minima and consequently, on helping it not to forget catastrophically. Expand
Lifelong Learning Without a Task Oracle
  • A. Rios, L. Itti
  • Computer Science
  • 2020 IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI)
  • 2020
TLDR
This work proposes and compares several candidate task-assigning mappers which require very little memory overhead and performs very close to a ground truth oracle, especially in experiments of inter-dataset task assignment. Expand
Organizing recurrent network dynamics by task-computation to enable continual learning
TLDR
A novel learning rule is developed designed to minimize interference between sequentially learned tasks in recurrent networks and it is shown that networks trained using this approach can reuse similar dynamical structures across similar tasks. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 78 REFERENCES
Overcoming catastrophic forgetting in neural networks
TLDR
It is shown that it is possible to overcome the limitation of connectionist models and train networks that can maintain expertise on tasks that they have not experienced for a long time and selectively slowing down learning on the weights important for previous tasks. Expand
Continual Lifelong Learning with Neural Networks: A Review
TLDR
This review critically summarize the main challenges linked to lifelong learning for artificial learning systems and compare existing neural network approaches that alleviate, to different extents, catastrophic forgetting. Expand
Continual Lifelong Learning with Neural Networks: A Review
TLDR
This review critically summarize the main challenges linked to lifelong learning for artificial learning systems and compare existing neural network approaches that alleviate, to different extents, catastrophic forgetting. Expand
Context-dependent computation by recurrent dynamics in prefrontal cortex
TLDR
This work studies prefrontal cortex activity in macaque monkeys trained to flexibly select and integrate noisy sensory inputs towards a choice, and finds that the observed complexity and functional roles of single neurons are readily understood in the framework of a dynamical process unfolding at the level of the population. Expand
An Empirical Investigation of Catastrophic Forgeting in Gradient-Based Neural Networks
TLDR
It is found that it is always best to train using the dropout algorithm--the drop out algorithm is consistently best at adapting to the new task, remembering the old task, and has the best tradeoff curve between these two extremes. Expand
Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem
TLDR
This chapter discusses the catastrophic interference in connectionist networks, and the simulation results demonstrate only that interference is catastrophic in some specific networks. Expand
Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization
TLDR
This study proposes a neuroscience-inspired scheme, called “context-dependent gating,” in which mostly nonoverlapping sets of units are active for any one task, which allows ANNs to maintain high performance across large numbers of sequentially presented tasks, particularly when combined with weight stabilization. Expand
Controlling Recurrent Neural Networks by Conceptors
TLDR
A mechanism of neurodynamical organization, called conceptors, is proposed, which unites nonlinear dynamics with basic principles of conceptual abstraction and logic, and helps explain how conceptual-level information processing emerges naturally and robustly in neural systems. Expand
How transferable are features in deep neural networks?
TLDR
This paper quantifies the generality versus specificity of neurons in each layer of a deep convolutional neural network and reports a few surprising results, including that initializing a network with transferred features from almost any number of layers can produce a boost to generalization that lingers even after fine-tuning to the target dataset. Expand
Connectionist models of recognition memory: constraints imposed by learning and forgetting functions.
  • R. Ratcliff
  • Computer Science, Medicine
  • Psychological review
  • 1990
TLDR
The problems discussed provide limitations on connectionist models applied to human memory and in tasks where information to be learned is not all available during learning. Expand
...
1
2
3
4
5
...