The role of natural language in a multimodal interface

  title={The role of natural language in a multimodal interface},
  author={Philip R. Cohen},
  booktitle={UIST '92},
Although graphics and direct manipulation are effective interface technologies for some classes of problems, they are limited in many ways. In particular, they provide little support for identifying objects not on the screen, for specifying temporal relations, for identifying and operating on large sets and subsets of entities, and for using the context of interaction. On the other hand, these are precisely strengths of natural language. This paper presents an interface that blends natural… 

Figures from this paper

Improving Information Search Using Natural Language and Direct Manipulation Tools in a Multimodal Interface
The usability of an existing, information-dense workspace, which places burdensome search requirements on its users, is evaluated and indicates that natural language is an effective search tool in a complex information workspace, because of its high level of specificity.
Eucalyptus: Integrating Natural Language Input with a Graphical User Interface
Eucalyptus, a natural language (NL) interface that has been integrated with the graphical user interface of the KOALAS Test Planning Tool, a simulated Naval air combat command system, handles both imperative commands and database queries while still allowing full use of the original graphical interface.
The history and future of direct manipulation
Recent developments in the implementation and understanding of direct manipulation interfaces point to various limitations in manipulative interaction which might be overcome through the selective reintroduction of conversational interaction.
Human-computer interaction through natural language and hypermedia in AlFresco
Two issues, important for a the more effective exploitation of the potentially large bandwidth of communication provided by this situation are presented: the integration of navigational and mediated aspects of interaction and the use of a graphical representation of the dialogue structure to allow the possibility of direct manipulation.
Extensible Multimodal Annotation for Intelligent Interactive Systems
This chapter describes the EMMA markup language and demonstrates its capabilities through presentation of a series of illustrative examples.
Multimodal Interaction for Information Access: Exploiting Cohesion
This work proposes use of an appropriate dialogue structure to constrain inference in the anaphora resolution process and allowance for the possibility of direct manipulation on this representation, to avoid the necessity of operating linguistically at the metalevel.
An improved interface for tutorial dialogues: browsing a visual dialogue history
This paper describes the design of a user interface that enables both the system and the user to refer to the past dialogue, based on the notion that the dialogue history is a source of knowledge that can be manipulated like any other.
Natural Language Access to Public Administration Data: The TAMIC-P System
Using Natural Language, as normally used between persons for communicating, reduces the skill requirements and enhances the system usability, showing that upon just a short training, a non-skilled operator is able to operate with effectiveness.
Multimedia Interfaces: Designing for Diversity
The interfaces commonly used now by computer users, the mouse, keyboard and screen, will be confined to those with desk jobs, and others will use a variety of input devices, from voice and pen to virtual reality interfaces.
HALPIN: A Natural Language Information Retrieval System for a Digital Library on the World Wide Web
The HALPIN system is developed to implement the multimodal conversational model for information retrieval in a digital library via the World Wide Web, which allows access to INRIA's database, on the Internet, in a natural language mode.


Synergistic use of direct manipulation and natural language
This paper shows how the integration of natural language with direct manipulation produces a multimodal interface that overcomes limitations of these techniques when used separately. Natural language
Direct Manipulation Interfaces
A cognitive account of both the advantages and disadvantages of direct manipulation interfaces is sought and two underlying phenomena that give rise to the feeling of directness of manipulation are identified.
Intelligent Multi-Media Interface Technology
A strategy to bring dialogue-like structure to user-system interaction is described, based on explicitly representing the "plans" of programs, and introducing those plans explicitly into the interface.
An Efficient Easily Adaptable System for Interpreting Natural Language Queries
This paper gives an overall account of a prototype natural language question answering system, called Chat-80. Chat-80 has been designed to be both efficient and easily adaptable to a variety of
Pointing: A Way Toward Explanation Dialogue
This paper proposes a hypertext-like interface that allows users to point to the portion of the system's explanation they would like clarified and employs a planning approach to explanation generation which records the design process that produced an explanation so that it can be used in later reasoning.
The Pragmatics of Referring and the Modality of Communication
It is concluded that intent recognition will need to be a central focus for pragmatics/discourse components of future speech understanding systems, and that computational linguistics needs to develop formalisms for reasoning about speakers' use of descriptions.
Direct Manipulation: A Step Beyond Programming Languages
As I talked with enthusiasts and examined the systems they used, I began to develop a model of the features that produced such delight, and the central ideas seemed to be visibility of the object of interest; rapid, reversible, incremental actions; and replacement of complex command language syntax by direct manipulation of the objects of interest.
Focusing and Description in Natural Language Dialogues
The relationship between focusing and definite descriptions in dialogue and its implications for natural language processing systems is examined and future research problems entailed in modeling the focusing process more generally are indicated.
Coping with Syntactic Ambiguity or How to Put the Block in the Box on the Table
This paper proposes some methods for dealing with syntactic ambiguity in ways that exploit certain regularities among alternative parse trees, and believes that such encoding of ambiguity will enhance processing, whether syntactic and semantic constraints are processed separately in sequence or interleaved together.