Between linguistic attention and gaze fixations inmultimodal conversational interfaces


In multimodal human machine conversation, successfully interpreting human attention is critical. While attention has been studied extensively in linguistic processing and visual processing, it is not clear how linguistic attention is aligned with visual attention in multimodal conversational interfaces. To address this issue, we conducted a preliminary investigation on how attention reflected by linguistic discourse aligns with attention indicated by gaze fixations during human machine conversation. Our empirical findings have shown that more attended entities based on linguistic discourse correspond to higher intensity of gaze fixations. The smoother a linguistic transition is, the less distance between corresponding fixation distributions. These findings provide insight into how language and gaze can be combined to predict attention, which have important implications in many tasks such as word acquisition and object recognition.

DOI: 10.1145/1647314.1647339

Extracted Key Phrases

13 Figures and Tables

Cite this paper

@inproceedings{Fang2009BetweenLA, title={Between linguistic attention and gaze fixations inmultimodal conversational interfaces}, author={Rui Fang and Joyce Yue Chai and Fernanda Ferreira}, booktitle={ICMI}, year={2009} }