Natural Language Direction Following for Robots in Unstructured Unknown Environments

  title={Natural Language Direction Following for Robots in Unstructured Unknown Environments},
  author={Felix Duvallet},
Abstract : Robots are increasingly performing collaborative tasks with people in homes, workplaces, and outdoors, and with this increase in interaction comes a need for e cient communication between human and robot teammates. One way to achieve this communication is through natural language, which provides a exible and intuitive way to issue commands to robots without requiring specialized interfaces or extensive user training. One task where natural language understanding could facilitate… 
Using Web Images & Natural Language for Object Localization in a Robotics Environment by
This work introduces a novel approach that allows robots to localize objects from an unbounded set of classes given only a description of a target object.
Using deep learning to bridge the gap between perception and intelligence
A method to for robot planning using deep learning to integrate object detection and natural language understanding by using a deep learning framework to replace learned interfaces with learned interfaces that select what data is shared between modules and allow for error back-propagation that could adapt each module to the robot’s task.
Learning of Complex-Structured Tasks from Verbal Instruction
This paper presents a novel approach to robot task learning from language-based instructions, which focuses on increasing the complexity of task representations that can be taught through verbal
Core Challenges in Embodied Vision-Language Planning
This paper proposes a taxonomy to unify Embodied Vision-Language Planning tasks, a family of prominent embodied navigation and manipulation problems that jointly use computer vision and natural language, and presents the core challenges that new EVLP works should seek to address and advocates for task construction that enables model generalizability and furthers real-world deployment.
placemat Place wine glass Place cup Place soda can Place spoon Place fork Place knife Place plate Place bowl
This paper presents a novel approach to robot task learning from language-based instructions, which focuses on increasing the complexity of task representations that can be taught through verbal
Incremental Semantic Mapping with Unsupervised On-line Learning
  • Y. C. SousaH. Bassani
  • Computer Science
    2018 International Joint Conference on Neural Networks (IJCNN)
  • 2018
This paper introduces an incremental semantic mapping approach, with on-line unsupervised learning, based on Self-Organizing Maps (SOM) for robotic agents, which demonstrates promising capabilities of incremental acquisition of topological maps enriched with semantic information.
Learning Multi-Modal Navigation for Unmanned Ground Vehicles
This work proposes a technique to transform a natural language command and a static aerial image, into a cost map suitable for planning, taking a step towards unifying language, perception, and planning in a single, end-to-end trainable system.


Imitation learning for natural language direction following through unknown environments
This work learns a policy which predicts a sequence of actions that follow the directions by exploring the environment and discovering landmarks, backtracking when necessary, and explicitly declaring when it has reached the destination.
Inferring Maps and Behaviors from Natural Language Instructions
This paper proposes a probabilistic framework that enables robots to follow commands given in natural language, without any prior knowledge of the environment, and demonstrates the algorithm’s ability to follow navigation commands with performance comparable to that of a fully-known environment.
Learning models for following natural language directions in unknown environments
A novel learning framework is proposed that enables robots to successfully follow natural language route directions without any previous knowledge of the environment by learning and performing inference over a latent environment model.
Toward Mobile Robots Reasoning Like Humans
An intelligence architecture that combines cognitive components to carry out high-level cognitive tasks, semantic perception to label regions in the world, and a natural language component to reason about the command and its relationship to the objects in theworld is developed.
Toward Information Theoretic Human-Robot Dialog
To enable a robot to recover from a failure to understand a natural language utterance, an information-theoretic strategy for asking targeted clarifying questions and using information from the answer to disambiguate the language is described.
Natural language command of an autonomous micro-air vehicle
This paper presents a micro-air vehicle capable of following natural language directions through a previously mapped and labeled environment, and demonstrates the robot following directions created by a human for another human, and interactively executing commands in the context of surveillance and search and rescue in confined spaces.
Interactive robot task training through dialog and demonstration
A framework for interactive task training of a mobile robot where the robot learns how to do various tasks while observing a human and how environmental context and communicative dialog with the human help the robot learn the task.
Learning to Parse Natural Language Commands to a Robot Control System
This work discusses the problem of parsing natural language commands to actions and control structures that can be readily implemented in a robot execution system, and learns a parser based on example pairs of English commands and corresponding control language expressions.
Learning to understand spatial language for robotic navigation and mobile manipulation
A method that uses previously detected objects and places in order to bias the search process toward areas of the environment where a previously unseen object is likely to be found and created on the fly as the robot explores the environment is presented.
A natural language planner interface for mobile manipulators
This paper presents a new model called the Distributed Correspondence Graph (DCG) to infer the most likely set of planning constraints from natural language instructions, and presents experimental results from comparative experiments that demonstrate improvements in efficiency in natural language understanding without loss of accuracy.