Semiotic schemas: A framework for grounding language in action and perception

@article{Roy2005SemioticSA,
  title={Semiotic schemas: A framework for grounding language in action and perception},
  author={Deb K. Roy},
  journal={Artif. Intell.},
  year={2005},
  volume={167},
  pages={170-205}
}
  • D. Roy
  • Published 1 September 2005
  • Computer Science
  • Artif. Intell.
Grounding words in perception and action: computational insights
  • D. Roy
  • Psychology
    Trends in Cognitive Sciences
  • 2005
Grounding words in perception and action : computational insights
We use words to communicate about things and kinds of things, their properties, relations and actions. Researchers are now creating robotic and simulated systems that ground language in machine
Grounding Words in Perception and Action : Insights from Computational Models
We use words to communicate about things and kinds of things, their properties, relations, and actions. Mirroring human abilities, researchers are creating robotic and simulated systems that ground
Grounding Words in Perception and Action: Insights from Computational Models
We use words to communicate about things and kinds of things, their properties, relations, and actions. Mirroring human abilities, researchers are creating robotic and simulated systems that ground
A Mechanistic Model of Three Facets of Meaning
This chapter presents a physical-computational model of sensory-motor grounded language interpretation for simple speech acts. The model is based on an implemented conversational robot. It combines a
Object schemas for grounding language in a responsive robot
An approach is introduced for physically grounded natural language interpretation by robots that reacts appropriately to unanticipated physical changes in the environment and dynamically assimilates
Situated Language Understanding as Filtering Perceived Affordances
TLDR
An implemented system that understands verbal commands situated in a virtual gaming environment that uses probabilistic hierarchical plan recognition to generate perceived affordances is described and quantitatively compares the system's predictions in response to direct player commands with the actions taken by human players.
Towards a layered framework for embodied language processing in situated human-robot interaction
We propose a layered architecture for embodied language processing in a robot, in which the language layer is grounded in a sensorimotor layer via a schematic representation layer, which represents
Vision and Action in the Language-Ready Brain: From Mirror Neurons to SemRep
TLDR
SemRep is presented, a graphical representation of dynamic visual scenes designed to support the generation of varied descriptions of episodes, and particular emphasis is placed on mirror neurons - neurons active both for execution of a certain class of actions and for recognition of a broader class of similar actions.
...
...

References

SHOWING 1-10 OF 95 REFERENCES
Grounding words in perception and action: computational insights
  • D. Roy
  • Psychology
    Trends in Cognitive Sciences
  • 2005
Grounding language in perception: A connectionist model of spatial terms and vegue quantifiers
TLDR
A new connectionist model of spatial language based on real psycholinguistic data puts together various constraints on object knowledge and on object localisation in order to influence the comprehension of a range of linguistic terms, mirroring what participants do in experiments.
Grounding language in action
TLDR
The data support an embodied theory of meaning that relates the meaning of sentences to human action, and are inconsistent with theories of language comprehension in which meaning is represented as a set of relations among nodes.
Coupling perception and simulation: steps towards conversational robotics
  • K. Hsiao, N. Mavridis, D. Roy
  • Computer Science, Art
    Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003) (Cat. No.03CH37453)
  • 2003
TLDR
A robotic system that uses a perceptually-coupled physical simulator to produce an internal world model of the robot's environment that allows the robot to be aware of objects no longer in its field of view (a form of "object permanence"), as well as to visualize its environment through the eyes of the user by enabling virtual shifts in point of view using synthetic vision operating within the simulator.
Mental imagery for a conversational robot
TLDR
A set of representations and procedures are presented that enable a robotic manipulator to maintain a "mental model" of its physical environment by coupling active vision to physical simulation and providing the basis for situated language comprehension and production.
Naive physics, event perception, lexical semantics, and language acquisition
TLDR
Algorithms are presented which utilize a cross-situational learning strategy whereby the learner finds a language model which is consistent across several utterances paired with their non-linguistic context, and three claims about event perception and the process of grounding language in visual perception are advanced.
Contentful mental states for robot baby
In this paper we claim that meaningful representations can be learned by programs, although today they are almost always designed by skilled engineers. We discuss several kinds of meaning that
Mental Models : Towards a Cognitive Science of Language
Mental Models offers nothing less than a unified theory of the major properties of mind: comprehension, inference, and consciousness. In spirited and graceful prose, Johnson-Laird argues that we
Understanding computers and cognition - a new foundation for design
TLDR
It is argued that what AI needs is not so much a hermeneutic approach as a better appreciation of biology and psychology.
The Altricial-Precocial Spectrum for Robots
TLDR
Consideration of the variety of capabilities and development patterns on the precocial-altricial spectrum in biological organisms will help to see high level methodological debates among AI researchers, linguists, psychologists and philosophers in a new light.
...
...