EMMT: A simultaneous eye-tracking, 4-electrode EEG and audio corpus for multi-modal reading and translation scenarios

@article{Bhattacharya2022EMMTAS,
  title={EMMT: A simultaneous eye-tracking, 4-electrode EEG and audio corpus for multi-modal reading and translation scenarios},
  author={Sunit Bhattacharya and Vvera Kloudov'a and Vil{\'e}m Zouhar and Ondvrej Bojar},
  journal={ArXiv},
  year={2022},
  volume={abs/2204.02905}
}
We present the Eyetracked Multi-Modal Translation (EMMT) corpus, a dataset containing monocular eye movement recordings, audio and 4-electrode electroencephalogram (EEG) data of 43 participants. The objective was to collect cognitive signals as responses of participants engaged in a number of language intensive tasks involving different text-image stimuli settings when translating from English to Czech. Each participant was exposed to 32 text-image stimuli pairs and asked to (1) read the… 
[PDF] Semantic Reader
3 Citations
Background Citations
2
Methods Citations
2

Figures and Tables from this paper

3 Citations

Multimodal Shannon Game with Images

It is shown that the addition of image information improves both self-reported confidence and accuracy for both humans and LM and highlights the potential of multimodal information in improving language understanding and modeling.

Multimodal Machines from a perspective of humans Ph.D Thesis Proposal

This thesis proposal focuses on highlighting the differences in the learning mechanisms of humans and deep learning systems and explores yet how recent work has established similarities between representations learnt by deeplearning systems and cognitive data collected from the human brain.

Sentence Ambiguity, Grammaticality and Complexity Probes

It is shown how features might be highly localized in the layers for these models and get lost in the upper layers and it is demonstrated that template-based datasets with surface-level artifacts should not be used for probing, careful comparisons with baselines should be done and that t-SNE plots should be used to determine the presence of a feature among dense vectors representations.

31 References

The Use of Electroencephalography in Language Production Research: A Review

It is concluded that overt speech production can be successfully studied using electrophysiological measures, for instance, event-related brain potentials (ERPs) and whether the ERP components in language production are comparable to results from other fields.

Choosing MUSE: Validation of a Low-Cost, Portable EEG System for ERP Research

It is demonstrated that with a single computer and a portable EEG system such as the MUSE one can conduct ERP research with ease thus greatly extending the possible use of the ERP methodology to a variety of novel contexts.

Event related potentials I

    R. Burgess
    Psychology, Biology
    Clinical Neurophysiology
  • 2001

Eye tracking as an MT evaluation technique

It is concluded that eye tracking has promise as a semi-automatic MT evaluation technique, which does not require bi-lingual knowledge, and which can potentially tap into the end users’ experience of machine translation output.

The ERP response to the amount of information conveyed by words in sentences

A brief introduction to the use of event-related potentials in studies of perception and attention.

    G. Woodman
    Psychology
    Attention, perception & psychophysics
  • 2010
A brief tutorial on the ERP technique is provided for consumers of such research and those considering the use of human electrophysiology in their own work.

A brief introduction to the use of event-related potentials in studies of perception and attention

Because of the precise temporal resolution of electrophysiological recordings, the event-related potential (ERP) technique has proven particularly valuable for testing theories of perception and…

Multimodal machine translation through visuals and speech

The paper concludes with a discussion of directions for future research in multimodal machine translation: the need for more expansive and challenging datasets, for targeted evaluations of model performance, and for multimodality in both the input and output space.

Assessing the Usability of Raw Machine Translated Output: A User-Centered Study Using Eye Tracking

This article reports on the results of a project that aimed to investigate the usability of raw machine translated technical support documentation for a commercial online file storage service, finding several significant differences between the source and MT output.

Eye Tracking as a Tool for Machine Translation Error Analysis

Overall the study shows that eye tracking can give complementary information to error analysis, such as aiding in ranking error types for seriousness, and correlations between eye tracking data and human estimates of text quality.