Learning Unknown Groundings for Natural Language Interaction with Mobile Robots

@inproceedings{Tucker2017LearningUG,
  title={Learning Unknown Groundings for Natural Language Interaction with Mobile Robots},
  author={Mycal Tucker and Derya Aksaray and Rohan Paul and Gregory J. Stein and Nicholas Roy},
  booktitle={International Symposium of Robotics Research},
  year={2017}
}
Our goal is to enable robots to understand or “ground” natural language instructions in the context of their perceived workspace. Contemporary models learn a probabilistic correspondence between input phrases and semantic concepts (or groundings) such as objects, regions or goals for robot motion derived from the robot’s world model. Crucially, these models assume a fixed and a priori known set of object types as well as phrases and train probable correspondences offline using static language… 
View via Publisher
dspace.mit.edu
21 Citations
Background Citations
13
Methods Citations
1

21 Citations

Grounding Natural Language Instructions with Unknown Object References using Learned Visual Attributes

This work considers the scenario of grounding unknown noun references in scenes populated multiple objects that the robot cannot recognize and builds on prior work and presents a model that resolves ambiguity amongst unknown groundings by eliciting visual attribute descriptions from the human.

Jointly Improving Parsing and Perception for Natural Language Commands through Human-Robot Dialog

Methods for using human-robot dialog to improve language understanding for a mobile robot agent that parses natural language to underlying semantic meanings and uses robotic sensors to create multi-modal models of perceptual concepts like red and heavy are presented.

Improving Grounded Natural Language Understanding through Human-Robot Dialog

This work presents an end-to-end pipeline for translating natural language commands to discrete robot actions, and uses clarification dialogs to jointly improve language parsing and concept grounding.

Language Understanding for Field and Service Robots in a Priori Unknown Environments

This paper provides a comprehensive description of a novel learning framework that allows field and service robots to interpret and correctly execute natural-language instructions in a priori unknown, unstructured environments and uses imitation learning to identify a belief-space policy that reasons over the environment and behavior distributions.

Multimodal estimation and communication of latent semantic knowledge for robust execution of robot instructions

A probabilistic model that fuses linguistic knowledge with visual and haptic observations into a cumulative belief over latent world attributes to infer the meaning of instructions and execute the instructed tasks in a manner robust to erroneous, noisy, or contradictory evidence is introduced.

Language-guided Semantic Mapping and Mobile Manipulation in Partially Observable Environments

A novel framework that learns to adapt perception according to the task in order to maintain compact distributions over semantic maps is proposed and experiments with a mobile manipulator demonstrate more efficient instruction following in a priori unknown environments.

Simultaneous Intention Estimation and Knowledge Augmentation via Human-Robot Dialog

A dialog agent for robots that is able to interpret user commands using a semantic parser, while asking clarification questions using a probabilistic dialog manager to augment its knowledge base and improve its language capabilities by learning from dialog experiences.

Augmenting Knowledge through Statistical, Goal-oriented Human-Robot Dialog

A dialog agent for robots that is able to interpret user commands using a semantic parser, while asking clarification questions using a probabilistic dialog manager to augment its knowledge base and improve its language capabilities by learning from dialog experiences.

Grounded Language Learning: Where Robotics and NLP Meet

An overview of the research area, selected recent advances, and some future directions and challenges that remain are given.

Sim-to-Real Transfer for Vision-and-Language Navigation

To bridge the gap between the high-level discrete action space learned by the VLN agent, and the robot's low-level continuous action space, a subgoal model is proposed to identify nearby waypoints, and domain randomization is used to mitigate visual domain differences.

References

SHOWING 1-10 OF 14 REFERENCES

Inferring Maps and Behaviors from Natural Language Instructions

This paper proposes a probabilistic framework that enables robots to follow commands given in natural language, without any prior knowledge of the environment, and demonstrates the algorithm’s ability to follow navigation commands with performance comparable to that of a fully-known environment.

Understanding Natural Language Commands for Robotic Navigation and Mobile Manipulation

A new model for understanding natural language commands given to autonomous systems that perform navigation and mobile manipulation in semi-structured environments that dynamically instantiates a probabilistic graphical model for a particular natural language command according to the command's hierarchical and compositional semantic structure.

Clarifying commands with information-theoretic human-robot dialog

An approach for enabling a robot to engage in clarifying dialog with a human partner, just as a human might do in a similar situation, is described, which improves the accuracy of command understanding over prior work while asking fewer questions as compared to baseline question-selection strategies.

Which one? Grounding the referent based on efficient human-robot interaction

This work proposes a set of strategies that allow a robot to identify the referent when the human partner refers to an object giving incomplete information, i.e. an ambiguous description, and proposes the use of an ontology to store and reason on the robot's knowledge to ease clarification, and therefore, improve interaction.

Reasoning about Unmodelled Concepts - Incorporating Class Taxonomies in Probabilistic Relational Models

This work proposes fuzzy inference in Markov logic networks, which enables the use of taxonomic knowledge as a source of imposing structure onto possible worlds and shows that by exploiting this structure, probability distributions can be represented more compactly and that the reasoning systems become capable of reasoning about concepts not contained in the probabilistic knowledge base.

Robot, asker of questions

Spoken Dialogue Management Using Probabilistic Reasoning

This work uses a Partially Observable Markov Decision Process (POMDP)-style approach to generate dialogue strategies by inverting the notion of dialogue state; the state represents the user's intentions, rather than the system state.

Edge Boxes: Locating Object Proposals from Edges

A novel method for generating object bounding box proposals using edges is proposed, showing results that are significantly more accurate than the current state-of-the-art while being faster to compute.

Efficient Grounding of Abstract Spatial Concepts for Natural Language Interaction with Robot Manipulators

This poster presents a probabilistic procedure to estimate the probability that a person’s immune system will be able to distinguish between a person wearing a robot and a person without a robot in the presence of a robot.

Vlfeat: an open and portable library of computer vision algorithms

VLFeat is an open and portable library of computer vision algorithms that includes rigorous implementations of common building blocks such as feature detectors, feature extractors, (hierarchical) k-means clustering, randomized kd-tree matching, and super-pixelization.