Distilling Neural Representations of Data Structure Manipulation using fMRI and fNIRS

@article{Huang2019DistillingNR,
  title={Distilling Neural Representations of Data Structure Manipulation using fMRI and fNIRS},
  author={Yu Huang and Xinyu Liu and Ryan Krueger and Tyler Santander and Xiaosu Hu and Kevin Leach and Westley Weimer},
  journal={2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)},
  year={2019},
  pages={396-407}
}
  • Yu Huang, Xinyu Liu, Westley Weimer
  • Published 25 May 2019
  • Biology, Psychology
  • 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)
Data structures permeate many aspects of software engineering, but their associated human cognitive processes are not thoroughly understood. We leverage medical imaging and insights from the psychological notion of spatial ability to decode the neural representations of several fundamental data structures and their manipulations. In a human study involving 76 participants, we examine list, array, tree, and mental rotation tasks using both functional near-infrared spectroscopy (fNIRS) and… 

Figures and Tables from this paper

Toward an Objective Measure of Developers’ Cognitive Activities
TLDR
Two controlled experiments involving 112 students to explore multiple computing activities (code comprehension, code review, and data structure manipulations) using three different objective measures including neuroimaging (functional near-infrared spectroscopy and functional magnetic resonance imaging (fMRI) and eye tracking) are presented.
Neurological Divide: An fMRI Study of Prose and Code Writing
TLDR
The first human study in which participants wrote code and prose while undergoing a functional magnetic resonance imaging (fMRI) brain scan, making use of a full-sized fMRI-safe QWERTY keyboard is presented.
Utilizing anatomical information for signal detection in functional magnetic resonance imaging
TLDR
This work proposes to define the regions for each subject separately based on their individual brain anatomy, represented, e.g., by so-called Aparc labels, and demonstrates that this approach can perform comparably to a two-stage approach for which two independent experiments are needed.
Comprehension of computer code relies primarily on domain-general executive brain regions
TLDR
Functional magnetic resonance imaging is used to investigate two candidate brain systems to investigate the multiple demand (MD) system and the language system, typically recruited during linguistic processing, and found that the MD system supports the use of novel cognitive tools even when the input is structurally similar to natural language.
Comprehension of computer code relies primarily on domain-general executive brain regions
Computer programming is a novel cognitive tool that has transformed modern society. What cognitive and neural mechanisms support this skill? Here, we used fMRI to investigate two candidate brain
Computer code comprehension shares neural resources with formal logical inference in the fronto-parietal network
TLDR
Code comprehension overlapped extensively with neural basis of formal logic and to a lesser degree math, and laterality of language and code covaried across individuals.
Computer code comprehension shares neural resources with formal logical inference in the fronto-parietal network
TLDR
In this study, expert programmers performed code comprehension and memory control tasks while undergoing fMRI, and the same participants also performed formal logic, symbolic math, executive control, and language localizer tasks.
Studying Programming in the Neuroage: Just a Crazy Idea?
TLDR
Using functional magnetic resonance imaging, scientists might better understand what goes on in the minds of programmers as they read and understand code through neuroimaging.
Understanding User Cognition: from Everyday Behavior and Spatial Ability to Code Writing and Review
Intellectual Merit. This thesis addresses how we can meaningfully and objectively measure cognitive processes in software engineering. We propose to use mobile crowdsensing, medical imaging and eye
Comprehension of computer code relies primarily on domain-general executive resources
TLDR
It is concluded that code comprehension relies primarily on domain-general executive resources, demonstrating that the multiple demand (MD) system supports the use of novel cognitive tools even when the input is structurally similar to natural language.
...
1
2
3
...

References

SHOWING 1-10 OF 96 REFERENCES
An fMRI analysis of the efficacy of Euler diagrams in logical reasoning
TLDR
This experiment showed that when Euler diagrams were present, response times in the task were significantly shorter than those in the usual reasoning task comprising only sentences, and it was suggested that complex processes of inferences can be replaced by simple diagram manipulation.
Changes in cortical activity during mental rotation. A mapping study using functional MRI.
TLDR
Functional MRI was used to observe focal changes in blood flow in the brains of 10 healthy volunteers performing a mental rotation task, and data are consistent with the hypothesis that mental rotation engages cortical areas involved in tracking moving objects and encoding spatial relations.
Model‐Based fMRI and Its Application to Reward Learning and Decision Making
TLDR
The approach of model‐based fMRI is summarized, with reference to the field of reward learning and decision making, where computational models have been used to probe the neural mechanisms underlying learning of reward associations, modifying action choice to obtain reward, as well as in encoding expected value signals that reflect the abstract structure of a decision problem.
fMRI Validation of fNIRS Measurements During a Naturalistic Task
TLDR
The results of the study show a task-related increase in oxyhemoglobin in both modalities and demonstrate that it is possible to replicate the findings of fMRI using fNIRS in a naturalistic task.
Decoding the Representation of Code in the Brain: An fMRI Study of Code Review and Expertise
TLDR
In a controlled experiment involving 29 participants, code comprehension, code review and prose review is examined using functional magnetic resonance imaging to find that the neural representations of programming languages vs. natural languages are distinct.
Developmental aspects of language processing: fMRI of verbal fluency in children and adults
TLDR
The brain areas that process semantic verbal fluency are similar in children and adults and the laterality of activation does not change appreciably with age and appears to be strongly lateralized by age 7 years.
Functional topography of the cerebellum for motor and cognitive tasks: An fMRI study
Different activation patterns for working memory load and visual attention load
...
1
2
3
4
5
...