Enabling Robots to Understand Incomplete Natural Language Instructions Using Commonsense Reasoning

@article{Chen2020EnablingRT,
  title={Enabling Robots to Understand Incomplete Natural Language Instructions Using Commonsense Reasoning},
  author={Haonan Chen and Hao Tan and Alan Kuntz and Mohit Bansal and Ron Alterovitz},
  journal={2020 IEEE International Conference on Robotics and Automation (ICRA)},
  year={2020},
  pages={1963-1969}
}
  • H. Chen, Hao Tan, +2 authors R. Alterovitz
  • Published 29 April 2019
  • Computer Science
  • 2020 IEEE International Conference on Robotics and Automation (ICRA)
Enabling robots to understand instructions provided via spoken natural language would facilitate interaction between robots and people in a variety of settings in homes and workplaces. However, natural language instructions are often missing information that would be obvious to a human based on environmental context and common sense, and hence does not need to be explicitly stated. In this paper, we introduce Language-Model-based Commonsense Reasoning (LMCR), a new method which enables a robot… Expand
Automated Assertion Generation from Natural Language Specifications
TLDR
This work implements a system that uses commonsense mappings to process input sentences into a normalized form and uses frame semantics to convert the normalized sentences into an IR based on the information and context contained in the Frames. Expand
Learning Instance-Level N-Ary Semantic Knowledge At Scale For Robots Operating in Everyday Environments
TLDR
This work proposes a transformer neural network that directly generalizes knowledge from observations of object instances that obtains a 10% improvement in predicting unknown properties of novel object instances while reducing training and inference time by 150 times. Expand
Talk-to-Resolve: Combining scene understanding and spatial dialogue to resolve granular task ambiguity for a collocated robot
TLDR
A system called Talk-to-Resolve (TTR) is presented that enables a robot to initiate a coherent dialogue exchange with the instructor by observing the scene visually to resolve the impasse and identifying the stalemate with 82% accuracy. Expand
Intent Recognition from Speech and Plan Recognition
TLDR
This paper proposes an algorithm to infer a speakers intentions with natural language analysis combined with plan recognition, and defines a Natural Language Understanding component to classify semantic roles from sentences into partially instantiated actions, that are interpreted as the intention of the speaker. Expand
Affordance Based Disambiguation and Validation in Human-Robot Dialogue
TLDR
This work presents an approach that tackles the validation of the resulting motions regarding feasibility by utilizing dynamic affordances for the disambiguation and elemental validation of speech-based instructions. Expand
Robot Sound Interpretation: Learning Visual-Audio Representations for Voice-Controlled Robots
TLDR
This work learns a representation that associates images and sound commands with minimal supervision and generates an intrinsic reward function to learn robotic tasks with reinforcement learning, and shows that this method outperforms previous work across various sound types and robotic tasks empirically. Expand
UNDERSTANDING MENTAL REPRESENTATIONS OF OBJECTS THROUGH VERBS APPLIED TO THEM
  • 2020
In order to interact with objects in our environment, we rely on an understanding of the actions that can be performed on them, and the extent to which they rely or have an effect on the propertiesExpand
Inference over Knowledge Representations Automatically Generated from Medical Texts with Applications in Healthcare Robotics
Many nations across the world, including the United States, face an impending shortage of trained medical professionals and personnel. The development of a robotic healthcare assistant would helpExpand
Learning and Reasoning for Robot Sequential Decision Making under Uncertainty
TLDR
This work proposes a robot SDM framework, called LCORPP, that supports the simultaneous capabilities of supervised learning for passive state estimation, automated reasoning with declarative human knowledge, and planning under uncertainty toward achieving long-term goals. Expand
Knowledge Acquisition and Reasoning Systems for Service Robots: A Short Review of the State of the Art
  • P. K. Prasad, W. Ertel
  • 2020 5th International Conference on Robotics and Automation Engineering (ICRAE)
  • 2020
Understanding the environment is an essential capability for mobile service robots to achieve autonomy. While this has proven to be a very challenging task, researchers have come up with a variety ofExpand
...
1
2
...

References

SHOWING 1-10 OF 104 REFERENCES
Inferring Maps and Behaviors from Natural Language Instructions
TLDR
This paper proposes a probabilistic framework that enables robots to follow commands given in natural language, without any prior knowledge of the environment, and demonstrates the algorithm’s ability to follow navigation commands with performance comparable to that of a fully-known environment. Expand
Improving Grounded Natural Language Understanding through Human-Robot Dialog
TLDR
This work presents an end-to-end pipeline for translating natural language commands to discrete robot actions, and uses clarification dialogs to jointly improve language parsing and concept grounding. Expand
Learning models for following natural language directions in unknown environments
TLDR
A novel learning framework is proposed that enables robots to successfully follow natural language route directions without any previous knowledge of the environment by learning and performing inference over a latent environment model. Expand
Tell me Dave: Context-sensitive grounding of natural language to manipulation instructions
TLDR
This paper presents a model that takes into account the variations in natural language and ambiguities in grounding them to robotic instructions with appropriate environment context and task constraints, based on an energy function that encodes such properties in a form isomorphic to a conditional random field. Expand
Understanding Natural Language Commands for Robotic Navigation and Mobile Manipulation
TLDR
This paper describes a new model for understanding natural language commands given to autonomous systems that perform navigation and mobile manipulation in semi-structured environments that dynamically instantiates a probabilistic graphical model for a particular natural language command according to the command's hierarchical and compositional semantic structure. Expand
Efficient grounding of abstract spatial concepts for natural language interaction with robot platforms
TLDR
Two new models for efficient natural language understanding of robot instructions are introduced, including a probabilistic model for interpreting abstract concepts that require hierarchical reasoning over constituent concrete entities as well as notions of cardinality and ordinality. Expand
A natural language planner interface for mobile manipulators
TLDR
This paper presents a new model called the Distributed Correspondence Graph (DCG) to infer the most likely set of planning constraints from natural language instructions, and presents experimental results from comparative experiments that demonstrate improvements in efficiency in natural language understanding without loss of accuracy. Expand
Grounding Abstract Spatial Concepts for Language Interaction with Robots
TLDR
A probabilistic model that incorporates an expressive space of abstract spatial concepts as well as notions of cardinality and ordinality is introduced that allows a robot to understand or “ground” natural language instructions in the context of its world model. Expand
Attention Based Natural Language Grounding by Navigating Virtual Environment
TLDR
An attention mechanism for multi-modal fusion of visual and textual modalities that allows the agent to learn to complete the task and achieve language grounding and it is shown that the learnt textual representations are semantically meaningful as they follow vector arithmetic in the embedding space. Expand
RoboFrameNet: Verb-centric semantics for actions in robot middleware
TLDR
Semantic frames are examined using semantic frames - a linguistics concept which describes scenes being acted out - as a conceptual stepping stone between natural language and robot action, and the scalability of this solution is examined through the development of RoboFrameNet. Expand
...
1
2
3
4
5
...