Low-Dimensional Structure in the Space of Language Representations is Reflected in Brain Responses
@article{Antonello2021LowDimensionalSI, title={Low-Dimensional Structure in the Space of Language Representations is Reflected in Brain Responses}, author={Richard J. Antonello and Javier Turek and Vy A. Vo and Alexander G. Huth}, journal={ArXiv}, year={2021}, volume={abs/2106.05426} }
How related are the representations learned by neural language models, translation models, and language tagging tasks? We answer this question by adapting an encoder-decoder transfer learning method from computer vision to investigate the structure among 100 different feature spaces extracted from hidden representations of various networks trained on language tasks. This method reveals a low-dimensional structure where language models and translation models smoothly interpolate between word…
10 Citations
Brain embeddings with shared geometry to artificial contextual embeddings, as a code for representing language in the human brain
- Computer SciencebioRxiv
- 2022
Using stringent, zero-shot mapping, it is demonstrated that brain embeddings in the IFG and the DLM contextual embedding space have strikingly similar geometry, which allows us to precisely triangulate the position of unseen words in both the brain embedding spaces.
Reconstructing the cascade of language processing in the brain using the internal computations of a transformer-based language model
- Computer SciencebioRxiv
- 2022
This paper decomposes the associated “transformations” into individual, functionally-specialized “attention heads” and demonstrates that the emergent syntactic computations performed by individual heads correlate with predictions of brain activity in specific cortical regions.
Neural Language Taskonomy: Which NLP Tasks are the most Predictive of fMRI Brain Activity?
- Psychology, Computer ScienceNAACL
- 2022
Transfer learning from representations learned for ten popular natural language processing tasks (two syntactic and eight semantic) for predicting brain responses from two diverse datasets: Pereira and Narratives.
A natural language fMRI dataset for voxelwise encoding models
- Computer Science, PsychologybioRxiv
- 2022
A dataset containing BOLD fMRI responses recorded while 8 subjects each listened to 27 complete, natural, narrative stories, accompanied by a python library containing basic code for creating voxelwise encoding models provides a large and novel resource for understanding speech and language processing in the human brain.
Self-supervised models of audio effectively explain human cortical responses to speech
- Computer Science, PsychologyICML
- 2022
Overall, these results show that self-supervised models effectively capture the hierarchy of information relevant to different stages of speech processing in human cortex.
Connecting Neural Response measurements & Computational Models of language: a non-comprehensive guide
- BiologyArXiv
- 2022
This survey traces a line from early research linking Event Related Potentials and complexity measures derived from simple language models to contemporary studies employing Artificial Neural Network models trained on large corpora in combination with neural response recordings from multiple modalities using naturalistic stimuli.
Toward a realistic model of speech processing in the brain with self-supervised learning
- BiologyArXiv
- 2022
The largest neuroimaging benchmark to date is shown, showing how self-supervised learning can account for a rich organization of speech processing in the brain, and thus delineate a path to identify the laws of language acquisition which shape the human brain.
Reprint: a randomized extrapolation based on principal components for data augmentation
- Computer ScienceSSRN Electronic Journal
- 2022
R EPRINT is appealing for its easy-to-use since it contains only one hyperparameter determining the dimension of subspace and requires low computational resource and suggests stable and consistent improvements in terms of suitable choices of principal components.
Deep Learning Models to Study Sentence Comprehension in the Human Brain
- Computer ScienceArXiv
- 2023
Recent artificial neural networks that
Multimodal foundation models are better simulators of the human brain
- Computer Science, BiologyArXiv
- 2022
It is proposed to explore the explainability of multimodal learning models with the aid of non-invasive brain imaging technologies such as functional magnetic resonance imaging (fMRI) and identify a number of brain regions where multimodally-trained encoders demonstrate better neural encoding performance.
References
SHOWING 1-10 OF 42 REFERENCES
Incorporating Context into Language Encoding Models for fMRI
- Computer Science, PsychologybioRxiv
- 2018
The models built here show a significant improvement in encoding performance relative to state-of-the-art embeddings in nearly every brain area and suggest that LSTM language models learn high-level representations that are related to representations in the human brain.
The neural architecture of language: Integrative reverse-engineering converges on a model for predictive processing
- Computer Science, Psychology
- 2020
Across models, significant correlations are observed among all three metrics of performance: neural fit, fit to behavioral responses, and accuracy on the next-word prediction task, consistent with the long-standing hypothesis that the brain’s language system is optimized for predictive processing.
Neural Taskonomy: Inferring the Similarity of Task-Derived Representations from Brain Activity
- Psychology, Computer SciencebioRxiv
- 2019
These computationally-driven results—arising out of state-of-the-art computer vision methods—begin to reveal the task-specific architecture of the human visual system.
Simultaneously Uncovering the Patterns of Brain Regions Involved in Different Story Reading Subprocesses
- PsychologyPloS one
- 2014
This approach is the first to simultaneously track diverse reading subprocesses during complex story processing and predict the detailed neural representation of diverse story features, ranging from visual word properties to the mention of different story characters and different actions they perform.
The Hierarchical Cortical Organization of Human Speech Processing
- PsychologyThe Journal of Neuroscience
- 2017
To investigate the processing steps performed by the human brain to transform natural speech sound into meaningful language, models based on a hierarchical set of speech features were used to predict BOLD responses of individual voxels recorded in an fMRI experiment while subjects listened to natural speech.
Task representations in neural networks trained to perform many cognitive tasks
- Psychology, BiologyNature Neuroscience
- 2019
It is found that after training, recurrent units can develop into clusters that are functionally specialized for different cognitive processes, and a simple yet effective measure is introduced to quantify relationships between single-unit neural representations of tasks.
Language Models are Unsupervised Multitask Learners
- Computer Science
- 2019
It is demonstrated that language models begin to learn these tasks without any explicit supervision when trained on a new dataset of millions of webpages called WebText, suggesting a promising path towards building language processing systems which learn to perform tasks from their naturally occurring demonstrations.
Topographic Mapping of a Hierarchy of Temporal Receptive Windows Using a Narrated Story
- PsychologyThe Journal of Neuroscience
- 2011
The results suggest that the time scale of processing is a functional property that may provide a general organizing principle for the human cerebral cortex.
Information-Theoretic Probing for Linguistic Structure
- Computer ScienceACL
- 2020
An information-theoretic operationalization of probing as estimating mutual information that contradicts received wisdom: one should always select the highest performing probe one can, even if it is more complex, since it will result in a tighter estimate, and thus reveal more of the linguistic information inherent in the representation.
Similarity of Neural Network Representations Revisited
- Computer ScienceICML
- 2019
A similarity index is introduced that measures the relationship between representational similarity matrices and does not suffer from this limitation of CCA.