We describe a corpus for research on learning everyday tasks in natural environments using the combination of natural language description and rich sensor data that we have collected for the CAET (Cognitive Assistant for Everyday Tasks) project. We have collected audio, video, Kinect RGB-Depth video and RFID object-touch data while participants demonstrate… (More)
We describe ongoing research towards building a cog-nitively plausible system for near one-shot learning of the meanings of attribute words and object names, by grounding them in a sensory model. The system learns incrementally from human demonstrations recorded with the Microsoft Kinect, in which the demonstrator can use unrestricted natural language… (More)
This Blue Sky presentation focuses on a major shift toward a notion of " ambient intelligence " that transcends general applications targeted at the general population. The focus is on highly personalized agents that accommodate individual differences and changes over time. This notion of Extended Ambient Intelligence (EAI) concerns adaptation to a person's… (More)
Analysis of the Symbol Grounding Problem has typically fo-cused on the nature of symbols and how they tie to perception without focusing on the actual qualities of what the symbols are to be grounded in. We formalize the requirements of the ground and propose a basic model of grounding perceptual primitives to regions in perceptual space that demonstrates… (More)
We describe a method for determining the names of RFID-tagged objects in activity videos using descriptions which have been parsed to provide anaphoric reference resolution and ontological categorization.
We approach the problem of understanding the disease progression and speech of patients suffering from ALS as a language divergence identification problem. We summarize the promises and challenges of using speech-related biomarkers to adapt speech recognition to ALS patients and correlate language divergence with disease progression.
Typically, visually-grounded language learning systems only accept feature data about objects in the environment that are explicitly mentioned, whether through annotation labels or direct reference through natural language. We show that when objects are described ambiguously using natural language, a system can use a combination of the pragmatic principles… (More)