Grounded Situation Models for Robots: Where words and percepts meet


Our long-term objective is to develop robots that engage in natural language-mediated cooperative tasks with humans. To support this goal, we are developing an amodal representation and associated processes which is called a grounded situation model (GSM). We are also developing a modular architecture in which the GSM resides in a centrally located module, around which there are language, perception, and action-related modules. The GSM acts as a sensor-updated "structured blackboard", that serves as a workspace with contents similar to a "theatrical stage" in the robot's "mind", which might be filled in with present, past or imagined situations. Two main desiderata drive the design of the GSM: first, "parsing" situations into ontological types and relations that reflect human language semantics, and second, allowing bidirectional translation between sensory-derived data/expectations and linguistic descriptions. We present an implemented system that allows of a range of conversational and assistive behavior by a manipulator robot. The robot updates beliefs (held in the GSM) about its physical environment, the human user, and itself, based on a mixture of linguistic, visual and proprioceptive evidence. It can answer basic questions about the present or past and also perform actions through verbal interaction. Most importantly, a novel contribution of our approach is the robot's ability for seamless integration of both language- and sensor-derived information about the situation: for example, the system can acquire parts of situations either by seeing them or by "imagining" them through descriptions given by the user: "There is a red ball at the left". These situations can later be used to create mental imagery and sensory expectations, thus enabling the aforementioned bidirectionality

DOI: 10.1109/IROS.2006.282258

Extracted Key Phrases

5 Figures and Tables


Citations per Year

103 Citations

Semantic Scholar estimates that this publication has 103 citations based on the available data.

See our FAQ for additional information.

Cite this paper

@article{Mavridis2006GroundedSM, title={Grounded Situation Models for Robots: Where words and percepts meet}, author={Nikolaos Mavridis and Deb Roy}, journal={2006 IEEE/RSJ International Conference on Intelligent Robots and Systems}, year={2006}, pages={4690-4697} }