Mechanisms of hierarchical reinforcement learning in cortico-striatal circuits 2: evidence from fMRI.

  title={Mechanisms of hierarchical reinforcement learning in cortico-striatal circuits 2: evidence from fMRI.},
  author={David Badre and Michael Joshua Frank},
  journal={Cerebral cortex},
  volume={22 3},
The frontal lobes may be organized hierarchically such that more rostral frontal regions modulate cognitive control operations in caudal regions. In our companion paper (Frank MJ, Badre D. 2011. Mechanisms of hierarchical reinforcement learning in corticostriatal circuits I: computational analysis. 22:509-526), we provide novel neural circuit and algorithmic models of hierarchical cognitive control in cortico-striatal circuits. Here, we test key model predictions using functional magnetic… 

Figures from this paper

Frontal cortex function derives from hierarchical predictive coding

By reconceptualizing lateral prefrontal activity as anticipating prediction errors, the HER model provides a novel unifying account of prefrontal cortex function with broad implications both for understanding the frontal cortex and building more powerful machine learning applications.

Is There Evidence for a Rostral-Caudal Gradient in Fronto-Striatal Loops and What Role Does Dopamine Play?

The hypothesis that dopamine may be an important neuromodulator in hierarchical processing, whereby frontal and striatal regions that have higher dopamine receptor density may have a larger influence over regions that exhibit lower dopamine receptordensity is formulated.

Reinforcement-learning in fronto-striatal circuits

It will be necessary to build bridges from algorithmic level descriptions of computational reinforcement-learning to implementational level models to better understand how reinforcement- learning emerges from multiple distributed neural networks in the brain.

Oscillatory dynamics coordinating human frontal networks in support of goal maintenance

The improved temporal resolution of human intracranial electrocorticography was utilized to investigate the mechanisms by which frontal cortical oscillatory networks communicate in support of hierarchical cognitive control, suggesting that inter-regional phase encoding is a mechanism for the dynamic instantiation of complex cognitive functions by frontal cortical subnetworks.

Converging Structural and Functional Connectivity of Orbitofrontal, Dorsolateral Prefrontal, and Posterior Parietal Cortex in the Human Striatum

A neurologically plausible network of converging corticostriatal projections that may support the integration of reward, executive control, and spatial attention that occurs during spatial reinforcement learning is delineated.

Hierarchical Error Representation: A Computational Model of Anterior Cingulate and Dorsolateral Prefrontal Cortex

The hierarchical error representation (HER) model of ACC/dlPFC suggests a mechanism by which hierarchically organized layers within ACC and dlPFC interact in order to solve sophisticated cognitive tasks.

Evidence for a Functional Hierarchy of Association Networks

The results indicate that hierarchical cognitive control is subserved by parallel and distributed association networks, together forming multiple localized functional gradients in different parts of association cortex.

Cognitive control over learning: creating, clustering, and generalizing task-set structure.

A new context-task-set (C-TS) model is developed, inspired by nonparametric Bayesian methods, that suggests that participants spontaneously build task-set structure into a learning problem when not cued to do so, and shows that C-TS provides a good quantitative fit to human sequences of choices.



Mechanisms of hierarchical reinforcement learning in corticostriatal circuits 1: computational analysis.

A hybrid Bayesian-reinforcement learning mixture of experts (MoE) model is developed, which can estimate the most likely hypothesis state of individual participants based on their observed sequence of choices and rewards.

Functional Magnetic Resonance Imaging Evidence for a Hierarchical Organization of the Prefrontal Cortex

Test whether the rostro-caudal axis of the PFC is organized hierarchically, based on the level of abstraction at which multiple representations compete to guide selection of action, finding that activation in PFC subregions was consistent with an abstract representational hierarchy.

Interactions between frontal cortex and basal ganglia in working memory: A computational model

In this model, the frontal cortex exhibits robust active maintenance, whereas the basal ganglia contribute a selective, dynamic gating function that enables frontal memory representations to be rapidly updated in a task-relevant manner.

Making Working Memory Work: A Computational Model of Learning in the Prefrontal Cortex and Basal Ganglia

This article presents an attempt to deconstruct this homunculus through powerful learning mechanisms that allow a computational model of the prefrontal cortex to control both itself and other brain areas in a strategic, task-appropriate manner.

Dopamine modulation in the basal ganglia locks the gate to working memory

A novel network model is presented to elucidate the role of phasic dopamine in the interaction of these two structures in initiating and maintaining mnemonic activity and suggests that neuromodulation plays a critical role in protecting memories against both internal and external sources of noise.

Testing the Reward Prediction Error Hypothesis with an Axiomatic Model

Measurements of human neural activity from the striatum, medial prefrontal cortex, amygdala, and posterior cingulate cortex satisfy necessary and sufficient conditions for the entire class of RPE models, but activity measured from the anterior insula falsifies the axiomatic model, and therefore no RPE model can account for measured activity.

Dynamic Dopamine Modulation in the Basal Ganglia: A Neurocomputational Account of Cognitive Deficits in Medicated and Nonmedicated Parkinsonism

A neural network model is presented that instantiates key biological properties and provides insight into the underlying role of DA in the BG during learning and execution of cognitive tasks and provides novel testable predictions for neuropsychological and pharmacological studies.

Neurocomputational models of basal ganglia function in learning, memory and choice

Hierarchical cognitive control deficits following damage to the human frontal lobe

Results provide direct evidence for a rostro-caudal hierarchical organization of the frontal lobes and two adjacent regions were distinguished on the basis of the level of control, consistent with previous functional magnetic resonance imaging results.