Learning offline: memory replay in biological and artificial reinforcement learning

  title={Learning offline: memory replay in biological and artificial reinforcement learning},
  author={Emma Roscow and Raymond Chua and Rui Ponte Costa and Matt W. Jones and Nathan F. Lepora},
  journal={Trends in Neurosciences},
Learning to act in an environment to maximise rewards is among the brain's key functions. This process has often been conceptualised within the framework of reinforcement learning, which has also gained prominence in machine learning and artificial intelligence (AI) as a way to optimise decision making. A common aspect of both biological and machine reinforcement learning is the reactivation of previously experienced episodes, referred to as replay. Replay is important for memory consolidation… Expand

Figures from this paper

20211027 Milstein offline memory replay
During spatial exploration, neural circuits in the hippocampus store memories of sequences of sensory events encountered in the environment. When sensory information is absent during “offline”Expand
Offline memory replay in recurrent neuronal networks emerges from constraints on online dynamics
The results demonstrate that biological constraints on the dynamics of recurrent neural circuits are sufficient to enable memories of sensory events stored in the strengths of synaptic connections to be flexibly read out during rest and sleep, which is thought to be important for memory consolidation and planning of future behavior. Expand


Brain-inspired replay for continual learning with artificial neural networks
A replay-based algorithm for deep learning without the need to store data is proposed in which internal or hidden representations are replayed that are generated by the network’s own, context-modulated feedback connections, and it provides a novel model for replay in the brain. Expand
Selective Experience Replay for Lifelong Learning
Overall, the results show that selective experience replay, when suitable selection algorithms are employed, can prevent catastrophic forgetting and is consistently the best approach on all domains tested. Expand
Continual Lifelong Learning with Neural Networks: A Review
This review critically summarize the main challenges linked to lifelong learning for artificial learning systems and compare existing neural network approaches that alleviate, to different extents, catastrophic forgetting. Expand
Human-level control through deep reinforcement learning
This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks. Expand
Overcoming catastrophic forgetting in neural networks
It is shown that it is possible to overcome the limitation of connectionist models and train networks that can maintain expertise on tasks that they have not experienced for a long time and selectively slowing down learning on the weights important for previous tasks. Expand
Continual Learning with Deep Generative Replay
The Deep Generative Replay is proposed, a novel framework with a cooperative dual model architecture consisting of a deep generative model ("generator") and a task solving model ("solver"), with only these two models, training data for previous tasks can easily be sampled and interleaved with those for a new task. Expand
Behavioural and computational evidence for memory consolidation biased by reward-prediction errors
This work provides a behavioural and theoretical toolkit with which to measure and interpret replay in striatal, hippocampal and neocortical circuits and disentangles the influences of salience on replay, suggesting that reinforcement learning is tuned by post-learning replay biased by reward-prediction error, not by reward per se. Expand
Toward an Integration of Deep Learning and Neuroscience
It is argued that a range of implementations of credit assignment through multiple layers of neurons are compatible with current knowledge of neural circuitry, and that the brain's specialized systems can be interpreted as enabling efficient optimization for specific problem classes. Expand
Prefrontal cortex as a meta-reinforcement learning system
A new theory is presented showing how learning to learn may arise from interactions between prefrontal cortex and the dopamine system, providing a fresh foundation for future research. Expand
Progressive Neural Networks
This work evaluates this progressive networks architecture extensively on a wide variety of reinforcement learning tasks, and demonstrates that transfer occurs at both low-level sensory and high-level control layers of the learned policy. Expand