• Corpus ID: 235390553

Low-Dimensional Structure in the Space of Language Representations is Reflected in Brain Responses

@article{Antonello2021LowDimensionalSI,
  title={Low-Dimensional Structure in the Space of Language Representations is Reflected in Brain Responses},
  author={Richard J. Antonello and Javier Turek and Vy A. Vo and Alexander G. Huth},
  journal={ArXiv},
  year={2021},
  volume={abs/2106.05426}
}
How related are the representations learned by neural language models, translation models, and language tagging tasks? We answer this question by adapting an encoder-decoder transfer learning method from computer vision to investigate the structure among 100 different feature spaces extracted from hidden representations of various networks trained on language tasks. This method reveals a low-dimensional structure where language models and translation models smoothly interpolate between word… 
[PDF] Semantic Reader
10 Citations
Highly Influential Citations
2
Background Citations
6
Methods Citations
2
Results Citations
1

Figures and Tables from this paper

10 Citations

Brain embeddings with shared geometry to artificial contextual embeddings, as a code for representing language in the human brain

Using stringent, zero-shot mapping, it is demonstrated that brain embeddings in the IFG and the DLM contextual embedding space have strikingly similar geometry, which allows us to precisely triangulate the position of unseen words in both the brain embedding spaces.

Reconstructing the cascade of language processing in the brain using the internal computations of a transformer-based language model

This paper decomposes the associated “transformations” into individual, functionally-specialized “attention heads” and demonstrates that the emergent syntactic computations performed by individual heads correlate with predictions of brain activity in specific cortical regions.

Neural Language Taskonomy: Which NLP Tasks are the most Predictive of fMRI Brain Activity?

Transfer learning from representations learned for ten popular natural language processing tasks (two syntactic and eight semantic) for predicting brain responses from two diverse datasets: Pereira and Narratives.

A natural language fMRI dataset for voxelwise encoding models

A dataset containing BOLD fMRI responses recorded while 8 subjects each listened to 27 complete, natural, narrative stories, accompanied by a python library containing basic code for creating voxelwise encoding models provides a large and novel resource for understanding speech and language processing in the human brain.

Self-supervised models of audio effectively explain human cortical responses to speech

Overall, these results show that self-supervised models effectively capture the hierarchy of information relevant to different stages of speech processing in human cortex.

Connecting Neural Response measurements & Computational Models of language: a non-comprehensive guide

This survey traces a line from early research linking Event Related Potentials and complexity measures derived from simple language models to contemporary studies employing Artificial Neural Network models trained on large corpora in combination with neural response recordings from multiple modalities using naturalistic stimuli.

Toward a realistic model of speech processing in the brain with self-supervised learning

The largest neuroimaging benchmark to date is shown, showing how self-supervised learning can account for a rich organization of speech processing in the brain, and thus delineate a path to identify the laws of language acquisition which shape the human brain.

Reprint: a randomized extrapolation based on principal components for data augmentation

R EPRINT is appealing for its easy-to-use since it contains only one hyperparameter determining the dimension of subspace and requires low computational resource and suggests stable and consistent improvements in terms of suitable choices of principal components.

Deep Learning Models to Study Sentence Comprehension in the Human Brain

Recent artificial neural networks that

Multimodal foundation models are better simulators of the human brain

It is proposed to explore the explainability of multimodal learning models with the aid of non-invasive brain imaging technologies such as functional magnetic resonance imaging (fMRI) and identify a number of brain regions where multimodally-trained encoders demonstrate better neural encoding performance.

References

SHOWING 1-10 OF 42 REFERENCES

Incorporating Context into Language Encoding Models for fMRI

The models built here show a significant improvement in encoding performance relative to state-of-the-art embeddings in nearly every brain area and suggest that LSTM language models learn high-level representations that are related to representations in the human brain.

The neural architecture of language: Integrative reverse-engineering converges on a model for predictive processing

Across models, significant correlations are observed among all three metrics of performance: neural fit, fit to behavioral responses, and accuracy on the next-word prediction task, consistent with the long-standing hypothesis that the brain’s language system is optimized for predictive processing.

Neural Taskonomy: Inferring the Similarity of Task-Derived Representations from Brain Activity

These computationally-driven results—arising out of state-of-the-art computer vision methods—begin to reveal the task-specific architecture of the human visual system.

Simultaneously Uncovering the Patterns of Brain Regions Involved in Different Story Reading Subprocesses

This approach is the first to simultaneously track diverse reading subprocesses during complex story processing and predict the detailed neural representation of diverse story features, ranging from visual word properties to the mention of different story characters and different actions they perform.

The Hierarchical Cortical Organization of Human Speech Processing

To investigate the processing steps performed by the human brain to transform natural speech sound into meaningful language, models based on a hierarchical set of speech features were used to predict BOLD responses of individual voxels recorded in an fMRI experiment while subjects listened to natural speech.

Task representations in neural networks trained to perform many cognitive tasks

It is found that after training, recurrent units can develop into clusters that are functionally specialized for different cognitive processes, and a simple yet effective measure is introduced to quantify relationships between single-unit neural representations of tasks.

Language Models are Unsupervised Multitask Learners

It is demonstrated that language models begin to learn these tasks without any explicit supervision when trained on a new dataset of millions of webpages called WebText, suggesting a promising path towards building language processing systems which learn to perform tasks from their naturally occurring demonstrations.

Topographic Mapping of a Hierarchy of Temporal Receptive Windows Using a Narrated Story

The results suggest that the time scale of processing is a functional property that may provide a general organizing principle for the human cerebral cortex.

Information-Theoretic Probing for Linguistic Structure

An information-theoretic operationalization of probing as estimating mutual information that contradicts received wisdom: one should always select the highest performing probe one can, even if it is more complex, since it will result in a tighter estimate, and thus reveal more of the linguistic information inherent in the representation.

Similarity of Neural Network Representations Revisited

A similarity index is introduced that measures the relationship between representational similarity matrices and does not suffer from this limitation of CCA.