Perceptual user interfaces: multimodal interfaces that process what comes naturally

@article{Oviatt2000PerceptualUI,
  title={Perceptual user interfaces: multimodal interfaces that process what comes naturally},
  author={Sharon L. Oviatt and Philip R. Cohen},
  journal={Commun. ACM},
  year={2000},
  volume={43},
  pages={45-53}
}
more transparent experience than ever before. Our voice, hands, and entire body, once augmented by sensors such as microphones and cameras, are becoming the ultimate transparent and mobile multimodal input devices. The area of multimodal systems has expanded rapidly during the past five years. Since Bolt’s [1] original “Put That There” concept demonstration, which processed speech and manual pointing during object manipulation, significant achievements have been made in developing more general… Expand
Multimodal Interfaces
  • S. Oviatt
  • Computer Science
  • Encyclopedia of Multimedia
  • 2008
TLDR
This chapter will review the main types of multimodal interfaces, their advantages and cognitive science underpinnings, primary features and architectural characteristics, and general research in the field of multi-modal interaction and interface design. Expand
Designing the User Interface for Multimodal Speech and Pen-Based Gesture Applications: State-of-the-Art Systems and Future Research Directions
TLDR
The emerging architectural approaches for interpreting speech and pen-based gestural input in a robust manner are summarized-including early and late fusion approaches, and the new hybrid symbolic-statistical approach. Expand
User-centered modeling and evaluation of multimodal interfaces
TLDR
This paper summarizes research on the cognitive science foundations of multimodal interaction, and on the essential role that user-centered modeling has played in prototyping, guiding, and evaluating the design of next-generation multi-modality interfaces, and describes the important role that selective methodologies and evaluation metrics have played in shaping next- generation multimodAL systems. Expand
Chapter 12 – Multimodal Input
TLDR
The focus is on the application of human cognitive processing models to improve understanding of multimodal behavior in different contexts, particularly in situations of high mental demand. Expand
Multimodal interaction: A review
  • M. Turk
  • Computer Science
  • Pattern Recognit. Lett.
  • 2014
TLDR
The question of early vs. late integration and inspiration in recent evidence in biological sensory integration are reviewed, as well as challenges that lie ahead for research in multimodal human-computer interaction. Expand
Combining Voice and Gesture for Human Computer Interaction
Recently, there has been a great deal of interest in multimodal interfaces thanks to their potential in providing more natural user-machine interactions, particularly in applications where the use ofExpand
Direct Touch Gaze Input Mid-Air Gestures Proxemics Wearable Speech Input
There has been a consistent push towards exploring novel input, display, and feedback technologies for sensemaking of data. However, most visual analytic systems in the wild that go beyond aExpand
Framing the Design Space of Multimodal Mid-Air Gesture and Speech-Based Interaction With Mobile Devices for Older People
TLDR
The aim of this work is to promote the usefulness and potential of multimodal technologies based on mid-air gestures and voice input for making older adults' interaction with mobile devices more accessible and inclusive. Expand
Natural User Interfaces
Computing devices have been changed and developed continuously since the early age of computing, and also many new kind of technology have been invented during that time. But user Interfaces, on theExpand
Learning to Interpret and Apply Multimodal Descriptions
TLDR
This dissertation concerns the task of learning to interpret multimodal descriptions composed of verbal utterances and hand gestures/sketches, and apply corresponding interpretations to tasks such as image retrieval. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 25 REFERENCES
Ten myths of multimodal interaction
TLDR
Well-designed multimodal systems integrate complementary modalities to yield a highly synergistic blend in which the strengths of each mode are capitalized upon and used to overcome weaknesses in the other. Expand
Designing the User Interface for Multimodal Speech and Pen-Based Gesture Applications: State-of-the-Art Systems and Future Research Directions
TLDR
The emerging architectural approaches for interpreting speech and pen-based gestural input in a robust manner are summarized-including early and late fusion approaches, and the new hybrid symbolic-statistical approach. Expand
Unification-based Multimodal Integration
TLDR
A multimodal language processing architecture which supports interfaces allowing simultaneous input from speech and gesture recognition is described, which allows the component modalities to mutually compensate for each others' errors. Expand
Multimodal interactive maps: designing for human performance
TLDR
In this research, interfaces supporting spoken, pen-based, and multimodal input were analyzed for their effectiveness in interacting with map systems and indicated that map displays can be structured to minimize performance errors and disfluencies effectively. Expand
“Put-that-there”: Voice and gesture at the graphics interface
  • R. Bolt
  • Computer Science
  • SIGGRAPH '80
  • 1980
TLDR
The work described herein involves the user commanding simple shapes about a large-screen graphics display surface, and because voice can be augmented with simultaneous pointing, the free usage of pronouns becomes possible, with a corresponding gain in naturalness and economy of expression. Expand
Mutual disambiguation of recognition errors in a multimodel architecture
TLDR
Although speech recognition as a stand-alone performed farmore poorly for accented speakers, their multimodal recognitionrates did not differ from those of native speakers, and implications are discussed for the development of future multi-modal architecture that can perform in a more robust and stable manner. Expand
Multimodal Integration - A Statistical View
TLDR
This work develops two techniques, an estimate approach and a learning approach, which are designed to optimize accurate recognition during the multimodal integration process, and evaluates these methods using Quickset, a speech/gesture multimodals system, and reports evaluation results based on an empirical corpus collected with Quicksets. Expand
Manual and gaze input cascaded (MAGIC) pointing
TLDR
This work explores a new direction in utilizing eye gaze for computer input by proposing an alternative approach, dubbed MAGIC (Manual And Gaze Input Cascaded) pointing, which might offer many advantages, including reduced physical effort and fatigue as compared to traditional manual pointing, greater accuracy and naturalness than traditional gaze pointing, and possibly fasterspeed than manual pointing. Expand
The efficiency of multimodal interaction: a case study
TLDR
A case study comparison of a directmanipulation-based graphical user interface (GUI) with the QuickSet pen/voice multi-modal interface for supporting the task of military force “laydown” suggests that there may be substantial efficiency advantages to multimodal interaction over GUIs for map-based tasks. Expand
QuickSet: multimodal interaction for distributed applications
QuickSet: Multimodal Interaction for Distributed Applications Philip R. Cohen, Michael Johnston, David McGee, Sharon Oviatt, Jay Pittman, Ira Smith, Liang Chen and Josh Glow Center for Human ComputerExpand
...
1
2
3
...