An extended reinforcement learning model of basal ganglia to understand the contributions of serotonin and dopamine in risk-based decision making, reward prediction, and punishment learning

@article{Balasubramani2014AnER,
  title={An extended reinforcement learning model of basal ganglia to understand the contributions of serotonin and dopamine in risk-based decision making, reward prediction, and punishment learning},
  author={Pragathi Priyadharsini Balasubramani and V. Srinivasa Chakravarthy and Balaraman Ravindran and Ahmed A. Moustafa},
  journal={Frontiers in Computational Neuroscience},
  year={2014},
  volume={8}
}
Although empirical and neural studies show that serotonin (5HT) plays many functional roles in the brain, prior computational models mostly focus on its role in behavioral inhibition. In this study, we present a model of risk based decision making in a modified Reinforcement Learning (RL)-framework. The model depicts the roles of dopamine (DA) and serotonin (5HT) in Basal Ganglia (BG). In this model, the DA signal is represented by the temporal difference error (δ), while the 5HT signal is… 

Figures and Tables from this paper

A network model of basal ganglia for understanding the roles of dopamine and serotonin in reward-punishment-risk based decision making
TLDR
The model predicts that optimizing 5HT levels along with DA medications might be essential for improving the patients' reward-punishment learning deficits, and describes how DA and 5HT mediate activity in these classes of neurons (D1R, D2R-, D1R-D2R- MSNs).
Modeling Serotonin’s Contributions to Basal Ganglia Dynamics
TLDR
This chapter presents an extended reinforcement learning (RL)-based model of DA and 5-HT function in the BG, which reconciles some of the diverse roles of 5- HT.
Neural circuit models of the serotonergic system
TLDR
A line of modelling that reconciles the diverse roles of serotonin in reward/punishment sensitivity, risk sensitivity and time-scale of reward integration, is described in detail.
Learning Reward Uncertainty in the Basal Ganglia
TLDR
A class of models that encode both the mean reward and the spread of the rewards, the former in the difference between the synaptic weights of D1 and D2 neurons, and the latter in their sum are presented.
Modeling Uncertainty-Seeking Behavior Mediated by Cholinergic Influence on Dopamine
TLDR
A leaky-integrate-and-fire model of the cholinergic system and its involvement in decision-making in particular is proposed, which implements a softmax-like selection with an uncertainty bonus by a Cholinergic drive to dopaminergic neurons, which in turn influence synaptic currents of downstream neurons.
Neural and neurochemical basis of reinforcement-guided decision making.
TLDR
Different as well as overlapping contributions of a large network of neural circuits and neurotransmitter systems to the processing of decision making are discussed, touching on neural circuitry and neuromodulatory regulation of exploratory decision making.
A closed loop framework of decision-making and learning in primate prefrontal circuits using Computational Modeling and Virtual Experimentation. (Un cadre en boucle fermée sur la prise de décision et l'apprentissage dans lescircuits préfrontaux des primates, par modélisation computationnelle et expé
TLDR
A computational systems-level framework that would help to develop an understanding of the organization of the prefrontal cortex (PFC) and the basal ganglia (BG) systems and their functional interactions in the process of decision-making and goal-directed behaviour in humans is built.
Identifying the Basal Ganglia Network Model Markers for Medication-Induced Impulsivity in Parkinson's Disease Patients
TLDR
Computational modeling has the potential to become an invaluable tool to predict the onset of behavioral changes during disease progression and a significant decrease in sensitivity to punishment and risk was crucial for explaining behavioral changes observed in PD-ON ICD patients.
The Protective Action Encoding of Serotonin Transients in the Human Brain
TLDR
Serotonergic concentrations transiently increase as a whole following negative reward prediction errors, while reversing when counterfactual losses predominate, which provides initial evidence that the serotonergic system acts as an opponent to dopamine signaling, as anticipated by theoretical models.
Coarse-Grained Neural Network Model of the Basal Ganglia to Simulate Reinforcement Learning Tasks
TLDR
The simplification of the CGNN model reduces its flexibility but improves the readability of the signal flow in comparison to more detailed FGNN models and can help to a greater extent in the translation between clinical neuroscience and computational modeling.
...
...

References

SHOWING 1-10 OF 110 REFERENCES
The Roles of Dopamine and Serotonin in Decision Making: Evidence from Pharmacological Experiments in Humans
  • R. Rogers
  • Psychology, Biology
    Neuropsychopharmacology
  • 2011
TLDR
It is suggested that the role played by serotonin in the regulation of cognitive biases, and representation of context in learning, point toward a role in the cortically mediated cognitive appraisal of reinforcers when selecting between actions, potentially accounting for its influence upon the processing salient aversive outcomes and social choice.
Serotonin Differentially Regulates Short- and Long-Term Prediction of Rewards in the Ventral and Dorsal Striatum
TLDR
The results suggest that serotonin controls the time scale of reward prediction by differentially regulating activities within the striatum.
What do the basal ganglia do? A modeling perspective
TLDR
The existing modeling literature on BG is reviewed and an integrative picture of the function of these nuclei is hypothesized and has the potential to radically alter treatment and management of BG-related neurological disorders and neuropsychiatric disorders also.
Acute Tryptophan Depletion in Healthy Volunteers Enhances Punishment Prediction but Does not Affect Reward Prediction
TLDR
The selective potentiation of punishment prediction concurs with recent theorizing, suggesting that central 5-HT carries a prediction error for future punishment, but not for future reward.
Serotonin and Dopamine: Unifying Affective, Activational, and Decision Functions
TLDR
Drawing on recent work on computational models of dopaminergic function to suggest a framework by which many of the seemingly diverse functions associated with both DA and serotonin—comprising both affective and activational ones, as well as a number of other functions not overtly related to either—can be seen as consequences of a single root mechanism.
...
...