Eye tracking as an MT evaluation technique

  title={Eye tracking as an MT evaluation technique},
  author={Stephen Doherty and Sharon O’Brien and Michael Carl},
  journal={Machine Translation},
Eye tracking has been used successfully as a technique for measuring cognitive load in reading, psycholinguistics, writing, language acquisition etc. [] Key Result We conclude that the eye tracking data, in particular gaze time and fixation count, correlate reasonably well with human evaluation of MT output but fixation duration and pupil dilation may be less reliable indicators of reading difficulty for MT output.
Eye Tracking as a Tool for Machine Translation Error Analysis
Overall the study shows that eye tracking can give complementary information to error analysis, such as aiding in ranking error types for seriousness, and correlations between eye tracking data and human estimates of text quality.
Assessing the Usability of Raw Machine Translated Output: A User-Centered Study Using Eye Tracking
This article reports on the results of a project that aimed to investigate the usability of raw machine translated technical support documentation for a commercial online file storage service, finding several significant differences between the source and MT output.
A cognitive perspective on equivalent effect: using eye tracking to measure equivalence in source text and target text cognitive effects on readers
The preliminary data indicate that this method can be used to compare the levels of cognitive effort between two readership groups in order to investigate whether their experience is similar – whether an ‘equivalent effect’ has been achieved – thus contributing to the ongoing discourse surrounding equivalence in translation studies.
The impact of translation modality on user experience: an eye-tracking study of the Microsoft Word user interface
The results suggest that language and translation modality play a significant role in the usability of software products whether users complete the given tasks or not and even if they are unaware that MT was used to translate the interface.
A review of eye-tracking applications as tools for training
Although limitations exist for using eye tracking as an interface tool, gaze and pupillary response have been successfully used to reflect changes in cognitive load and are starting to be incorporated into adaptive training systems, although issues are still present with differentiating pupil responses from simultaneous psychological effects.
A Pilot Eye-Tracking Study of WMT-Style Ranking Evaluation
This paper describes the first steps towards explaining the scoring task: it runs the scoring under an eye-tracker and monitor what the annotators do, and tests the feasibility of eye tracking for the analysis of such a complex MT evaluation setup.
JEZIK: A Cognitive Translation System Employing a Single, Visible Spectrum Tracking Detector
A cognitive translation system has been built employing a single, inexpensive web camera without the use of infrared illumination and it was shown that the system translates the text, detects rarely occurring and out-of-context words from eye gaze information, and provides solutions in real time while the user is still reading.
Investigating the effects of controlled language on the reading and comprehension of machine translated texts: A mixed-methods approach
The research finds that the use of CL input results in significantly higher scores for items recalled by participants, and for several of the eye tracking metrics: fixation count, fixation length, and regressions.
GECO-MT: The Ghent Eye-tracking Corpus of Machine Translation
A large corpus of eye movement data, collected during natural reading of a human translation and a machine translation of a full novel, expands upon an earlier corpus called GECO (Ghent Eye-tracking Corpus of Machine Translation), and is described.
Applications of Eye Tracking in Language Processing and Other Areas
This chapter presents an overview on applications of eye-tracking technology in natural language processing (NLP) and other areas and summarizes various methods proposed to include eye-movement data in different components of NLP pipeline viz. annotation, classification, and evaluation.


The results demonstrate that eye‐tracking data, including pupil dilation measurements and gaze replays, in association with retrospective protocols promise to be a very effective methodology for future research into translation processes.
Processing fuzzy matches in Translation Memory tools : an eye-tracking analysis
Little research has been carried out to date on how translators interact with Translation Memory (TM) tools. In particular, there has been very little research on the effort required to process
Using pupillometric, fixation-based and subjective measures to measure the processing effort experienced when viewing subtitled TV anime with pop-up gloss
Eye movements and pupil size of twenty participants were recorded while they were watching excerpts from a TV anime subtitled in english with either standard subtitling, or subtitling that included
Do adult readers know how they read? Evidence from eye movement patterns and verbal reports.
It is concluded that systematic and extensive looking back in text is indicative of strategic behaviour and adult readers are well aware of their general reading speed and reasonablyaware of their lookback and rereading behaviour.
The act of task difficulty and eye-movement frequency for the 'Oculo-motor indices'
The results suggests that oculo-motor indices respond to task dif culty, and eye-movement frequency appears to play a different role than that of task d if culty.
Eye movement behaviour across four different types of reading task
It was found that professionals were faster than students in the final task, and it was shown that the distribution of visual attention to the source text for students was higher than that for the target text, whereas professional translators prioritised visual Attention to their own target text.
Eye movements in reading and information processing: 20 years of research.
  • K. Rayner
  • Psychology
    Psychological bulletin
  • 1998
The basic theme of the review is that eye movement data reflect moment-to-moment cognitive processes in the various tasks examined.
Perspective Effects on Expository Text Comprehension: Evidence From Think-Aloud Protocols, Eyetracking, and Recall
In this study, 36 participants read an expository text describing 4 rare illnesses from a given perspective. Their eye movements were recorded during reading, and think-alouds were probed after 10
Individual differences in working memory and reading
How prior knowledge, WMC, and relevance of information affect eye fixations in expository text.
The results suggest that if the reader has prior knowledge related to text contents and a high WMC, relevant text information can be encoded into memory without extra processing time.