Eligibility Traces and Plasticity on Behavioral Time Scales: Experimental Support of NeoHebbian Three-Factor Learning Rules

@article{Gerstner2018EligibilityTA,
  title={Eligibility Traces and Plasticity on Behavioral Time Scales: Experimental Support of NeoHebbian Three-Factor Learning Rules},
  author={Wulfram Gerstner and Marco P Lehmann and Vasiliki Liakoni and Dane S. Corneil and Johanni Brea},
  journal={Frontiers in Neural Circuits},
  year={2018},
  volume={12}
}
Most elementary behaviors such as moving the arm to grasp an object or walking into the next room to explore a museum evolve on the time scale of seconds; in contrast, neuronal action potentials occur on the time scale of a few milliseconds. Learning rules of the brain must therefore bridge the gap between these two different time scales. Modern theories of synaptic plasticity have postulated that the co-activation of pre- and postsynaptic neurons sets a flag at the synapse, called an… 

Figures from this paper

Behavioral Time Scale Plasticity of Place Fields: Mathematical Analysis
TLDR
These experimental results suggest that presynaptic activity generates synaptic eligibility traces both for potentiation and depression, which last on the order of seconds, and a simple mathematical model is developed that is consistent with these observations.
Sequence anticipation and STDP emerge from a voltage-based predictive learning rule
TLDR
A plasticity rule based on pre-dictive processing is proposed, where the neuron learns a low-rank model of the synaptic input dynamics in its membrane potential, which can explain the development of anticipatory motion signaling and recall in the visual system.
Synaptic Plasticity Forms and Functions.
TLDR
It is suggested that the effort to determine the neural basis of adaptive behavior could benefit from renewed experimental and theoretical investigation of more powerful directed types of synaptic plasticity.
A behavioural correlate of the synaptic eligibility trace in the nucleus accumbens
TLDR
Findings support the application of the synaptic eligibility trace to construct biologically plausible RL models by finding that the reward-sensitive period was within 1 s after the pure tone presentation and optogenetically-induced presynaptic activities at the NAc.
A Computational Theory of Learning Flexible Reward-Seeking Behavior with Place Cells
TLDR
A computational theory that achieves behavioral flexibility with better biological plausibility is proposed and implemented into a high-fidelity virtual rat in the MuJoCo physics simulator, showing significantly better learning efficiency and behavioral flexibility than a rat that implements a neuroscience-inspired reinforcement learning algorithm, deep Q-network.
The minimal behavioral time window for reward conditioning in the nucleus accumbens of mice
TLDR
The data suggest that the minimal reward time window is 1 s, and is formed in the nucleus accumbens, which is in agreement with previous reports on the effective dopamine timing of NAc synapses.
Norepinephrine potentiates and serotonin depresses visual cortical responses by transforming eligibility traces
TLDR
The induction of eTraces and their conversion to LTP/D by norepinephrine and serotonin respectively potentiates and depresses visual responses and is crucial for ocular dominance plasticity, a canonical model of experience-dependent plasticity.
Norepinephrine potentiates and serotonin depresses visual cortical responses by transforming eligibility traces
TLDR
It is shown in vivo in V1 that the induction of eTraces and their conversion to LTP/D by norepinephrine and serotonin respectively potentiates and depresses visual responses and is crucial for ocular dominance plasticity, a canonical model of experience-dependent plasticity.
Remembrance of things practiced: A two-pathway circuit for sequential learning
TLDR
Combining error-based and associative learning, this work mathematically derive the forgetting curve for a single neuron and its dependence on practiced repetition, showing how highly practiced memories or behaviors can become far more resistant to being overwritten by later learning.
Modulation of Spike-Timing Dependent Plasticity: Towards the Inclusion of a Third Factor in Computational Models
TLDR
Elucidation of the mechanisms underlying STDP modulation and its consequences on network dynamics is of critical importance and will allow better understanding of the major mechanisms of memory storage and recall both in health and disease.
...
...

References

SHOWING 1-10 OF 199 REFERENCES
Neuromodulated Spike-Timing-Dependent Plasticity, and Theory of Three-Factor Learning Rules
TLDR
The importance of synaptic mechanisms in bridging the temporal gap between sensory stimulation and neuromodulatory signals is highlighted, and a framework for a class of neo-Hebbian three-factor learning rules that depend on presynaptic activity, postsynaptic variables as well as the influence of neurmodulators is developed.
Prospective Coding by Spiking Neurons
TLDR
A biologically plausible synaptic plasticity rule is proposed to learn predictions on a single neuron level on a timescale of seconds and the potential role of the learning mechanism in classical trace conditioning is discussed.
Solving the distal reward problem through linkage of STDP and dopamine signaling
TLDR
This study emphasizes the importance of precise firing patterns in brain dynamics and suggests how a global diffusive reinforcement signal in the form of extracellular DA can selectively influence the right synapses at the right time.
Distinct Eligibility Traces for LTP and LTD in Cortical Synapses
Sequential neuromodulation of Hebbian plasticity offers mechanism for effective reward-based navigation
TLDR
It is demonstrated that sequential neuromodulation of STDP by acetylcholine and dopamine offers an efficacious model of reward-based navigation, and also provides a possible mechanism for aligning the time scales of cellular and behavioral learning.
Behavioral time scale synaptic plasticity underlies CA1 place fields
TLDR
It is discovered that place fields in hippocampal area CA1 are produced by a synaptic potentiation notably different from Hebbian plasticity, which abruptly modifies inputs that were neither causal nor close in time to postsynaptic activation.
A neuronal learning rule for sub-millisecond temporal coding
TLDR
A modelling study based on computer simulations of a neuron in the laminar nucleus of the barn owl shows that the necessary degree of coherence in the signal arrival times can be attained during ontogenetic development by virtue of an unsupervised hebbian learning rule.
A critical time window for dopamine actions on the structural plasticity of dendritic spines
TLDR
A molecular basis of reinforcement plasticity at the level of single dendritic spines is described, which could provide insight into psychiatric disorders involving dopaminergic regulation, such as depression, drug addiction, and schizophrenia.
Multicontact Co-operativity in Spike-Timing–Dependent Structural Plasticity Stabilizes Networks
TLDR
A combined model of structural and spike-timing–dependent plasticity that explains the multicontact configuration of synapses in adult neocortical networks under steady-state and lesion-induced conditions and finds that co-operation of multiple synaptic contacts is crucial for stable, long-term synaptic memories.
Reinforcement Learning Through Modulation of Spike-Timing-Dependent Synaptic Plasticity
TLDR
It is shown that the modulation of STDP by a global reward signal leads to reinforcement learning, and analytically learning rules involving reward-modulated spike-timing-dependent synaptic and intrinsic plasticity are derived, which may be used for training generic artificial spiking neural networks, regardless of the neural model used.
...
...