Comparing continual task learning in minds and machines

@article{Flesch2018ComparingCT,
  title={Comparing continual task learning in minds and machines},
  author={Timo Flesch and Jan Balaguer and Ronald Dekker and Hamed Nili and Christopher Summerfield},
  journal={Proceedings of the National Academy of Sciences of the United States of America},
  year={2018},
  volume={115},
  pages={E10313 - E10322}
}
Significance Humans learn to perform many different tasks over the lifespan, such as speaking both French and Spanish. The brain has to represent task information without mutual interference. In machine learning, this “continual learning” is a major unsolved challenge. Here, we studied the patterns of errors made by humans and state-of-the-art neural networks while they learned new tasks from scratch and without instruction. Humans, but not machines, seem to benefit from training regimes that… 

Figures from this paper

Towards continual task learning in artificial neural networks: current approaches and insights from neuroscience
TLDR
The influence of network architecture on task performance and representational interference will be addressed, providing a broad critical appraisal of current approaches to continual learning, while interrogating the extent to which insight might be provided by the rich literature of learning and memory in neuroscience.
A Neural Network Model of Continual Learning with Cognitive Control
TLDR
It is shown that neural networks equipped with a mechanism for cognitive control do not exhibit catastrophic forgetting when trials are blocked, and an advantage of blocking over interleaving when there is a bias for active maintenance in the control signal, implying a tradeoff between maintenance and the strength of control.
Computational evidence for hierarchically structured reinforcement learning in humans
TLDR
Evidence is presented that hierarchical RL, a biologically inspired and computationally simple algorithm, can capture human behavior in complex, hierarchical environments and opens the avenue for future research in this field.
Using top-down modulation to optimally balance shared versus separated task representations
TLDR
This work identifies and systematically evaluates two crucial features of modulatory signals, resulting in four modulation networks which are tested on a variety of input datasets and tasks with different degrees of stimulus-action mapping overlap.
Multi-task reinforcement learning in humans.
TLDR
Across three exploratory experiments and a large preregistered confirmatory experiment, the results provide evidence that participants who are able to learn the task use a strategy that maps previously learned policies to novel scenarios, enriching the understanding of human reinforcement learning in complex environments with changing task demands.
Rich and lazy learning of task representations in brains and neural networks
TLDR
Evidence is reported for neural coding patterns in biological brains whose dimensionality and neural geometry are consistent with the rich learning regime, using behavioural testing and neuroimaging in humans and analysis of neural signals from macaque prefrontal cortex.
Multi-Task Reinforcement Learning in Humans
TLDR
Across three exploratory experiments and a large preregistered experiment, the results provide strong evidence for a strategy that maps previously learned policies to novel scenarios that enrich the understanding of human reinforcement learning in complex environments with changing task demands.
Generalizing to generalize: Humans flexibly switch between compositional and conjunctive structures during reinforcement learning
TLDR
It is found that human generalization is sensitive to the statistics of the previously experienced task domain, favoring compositional or conjunctive generalization when the task statistics are indicative of such structures, and a mixture of the two when they are more ambiguous.
Maslow's Hammer for Catastrophic Forgetting: Node Re-Use vs Node Activation
TLDR
This paper theoretically analyse both a synthetic teacher-student framework and a real data setup to provide an explanation of the presence of a trade-off between node activation and node re-use that results in worst forgetting in the intermediate regime.
Learning to Read through Machine Teaching
TLDR
This work used a cognitively interesting neural network architecture to examine whether the sequence of learning trials could be structured to facilitate learning and observed significant improvement on generalization accuracy compared to baseline conditions.
...
...

References

SHOWING 1-10 OF 51 REFERENCES
Continual Lifelong Learning with Neural Networks: A Review
Overcoming catastrophic forgetting in neural networks
TLDR
It is shown that it is possible to overcome the limitation of connectionist models and train networks that can maintain expertise on tasks that they have not experienced for a long time and selectively slowing down learning on the weights important for previous tasks.
Cognitive control over learning: creating, clustering, and generalizing task-set structure.
TLDR
A new context-task-set (C-TS) model is developed, inspired by nonparametric Bayesian methods, that suggests that participants spontaneously build task-set structure into a learning problem when not cued to do so, and shows that C-TS provides a good quantitative fit to human sequences of choices.
Human category learning.
TLDR
Results from four different kinds of category-learning tasks provide strong evidence that human category learning is mediated by multiple, qualitatively distinct systems.
The Cost of Structure Learning
  • A. Collins
  • Psychology
    Journal of Cognitive Neuroscience
  • 2017
TLDR
These findings confirm the theory that humans pay a high short-term cost in learning structure to enable longer-term benefits in learning flexibility, and supported the prediction that structure learning incurred a major learning cost and was specifically tied to the effort in selecting abstract rules, leading to more errors when applying those rules.
Catastrophic forgetting in connectionist networks
  • R. French
  • Computer Science
    Trends in Cognitive Sciences
  • 1999
Human-level control through deep reinforcement learning
TLDR
This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks.
Learning bundles of stimuli renders stimulus order as a cue, not a confound
  • Ting Qian, R. Aslin
  • Psychology, Biology
    Proceedings of the National Academy of Sciences
  • 2014
TLDR
It is argued that the perspective of viewing stimulus order as a cue may hold the key to explaining behaviors that seemingly deviate from normative theories of cognition and that in task domains where the assumption of stimulus bundles is intuitively appropriate, it can improve the explanatory power of existing models.
The importance of mixed selectivity in complex cognitive tasks
TLDR
It is shown that mixed selectivity neurons encode distributed information about all task-relevant aspects, so that each aspect can be decoded from the population of neurons even when single-cell selectivity to that aspect is eliminated.
Connectionist models of recognition memory: constraints imposed by learning and forgetting functions.
  • R. Ratcliff
  • Psychology, Computer Science
    Psychological review
  • 1990
TLDR
The problems discussed provide limitations on connectionist models applied to human memory and in tasks where information to be learned is not all available during learning.
...
...