Natural Language Direction Following for Robots in Unstructured Unknown Environments
@inproceedings{Duvallet2015NaturalLD,
title={Natural Language Direction Following for Robots in Unstructured Unknown Environments},
author={Felix Duvallet},
year={2015}
}Abstract : Robots are increasingly performing collaborative tasks with people in homes, workplaces, and outdoors, and with this increase in interaction comes a need for e cient communication between human and robot teammates. One way to achieve this communication is through natural language, which provides a exible and intuitive way to issue commands to robots without requiring specialized interfaces or extensive user training. One task where natural language understanding could facilitate…
Figures and Tables from this paper
figure 1.1 
figure 1.2 
table 2.1 
figure 3.1 
table 3.2 
figure 3.2 
table 3.3 
figure 3.3 
figure 3.4 
figure 3.5 
figure 3.6 
figure 3.7 
figure 3.8 
figure 4.1 
table 4.1 
figure 4.2 
figure 4.2 
table 4.2 
figure 4.3 
figure 4.4 
figure 4.5 
figure 4.6 
figure 4.7 
figure 4.8 
figure 4.9 
figure 5.1 
table 5.1 
figure 5.2 
table 5.2 
figure 5.3 
figure 5.4 
figure 5.5 
figure 5.6 
figure 6.1 
figure 6.2 
figure 6.3 
figure 6.4 
figure 6.5 
figure 6.6 
figure 6.7 
figure 7.1 
table 7.1 
figure 7.2 
table 7.2 
figure 7.3 
table 7.3 
figure 7.4 
table 7.4 
figure 7.5 
table 7.5 
figure 7.6 
figure 7.7 
figure 7.8 
figure 7.9 
figure 8.1 
figure 8.2 
table 8.2 
figure 8.3 
figure 8.4 
figure 8.5 
figure 8.6 
figure 8.7 
figure A.1 
figure A.2 
figure A.3 
figure A.4 
figure A.5
8 Citations
Using Web Images & Natural Language for Object Localization in a Robotics Environment by
- Computer Science
- 2021
This work introduces a novel approach that allows robots to localize objects from an unbounded set of classes given only a description of a target object.
Using deep learning to bridge the gap between perception and intelligence
- Computer ScienceDefense + Security
- 2017
A method to for robot planning using deep learning to integrate object detection and natural language understanding by using a deep learning framework to replace learned interfaces with learned interfaces that select what data is shared between modules and allow for error back-propagation that could adapt each module to the robot’s task.
Learning of Complex-Structured Tasks from Verbal Instruction
- Computer Science2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids)
- 2019
This paper presents a novel approach to robot task learning from language-based instructions, which focuses on increasing the complexity of task representations that can be taught through verbal…
Core Challenges in Embodied Vision-Language Planning
- Computer ScienceJ. Artif. Intell. Res.
- 2022
This paper proposes a taxonomy to unify Embodied Vision-Language Planning tasks, a family of prominent embodied navigation and manipulation problems that jointly use computer vision and natural language, and presents the core challenges that new EVLP works should seek to address and advocates for task construction that enables model generalizability and furthers real-world deployment.
placemat Place wine glass Place cup Place soda can Place spoon Place fork Place knife Place plate Place bowl
- Computer Science
- 2019
This paper presents a novel approach to robot task learning from language-based instructions, which focuses on increasing the complexity of task representations that can be taught through verbal…
Opportunity and risk are twins. An investigation into algorithms.
- Economics
- 2019
Department of Economics
Incremental Semantic Mapping with Unsupervised On-line Learning
- Computer Science2018 International Joint Conference on Neural Networks (IJCNN)
- 2018
This paper introduces an incremental semantic mapping approach, with on-line unsupervised learning, based on Self-Organizing Maps (SOM) for robotic agents, which demonstrates promising capabilities of incremental acquisition of topological maps enriched with semantic information.
Learning Multi-Modal Navigation for Unmanned Ground Vehicles
- Computer Science
- 2018
This work proposes a technique to transform a natural language command and a static aerial image, into a cost map suitable for planning, taking a step towards unifying language, perception, and planning in a single, end-to-end trainable system.
References
SHOWING 1-10 OF 186 REFERENCES
Imitation learning for natural language direction following through unknown environments
- Computer Science2013 IEEE International Conference on Robotics and Automation
- 2013
This work learns a policy which predicts a sequence of actions that follow the directions by exploring the environment and discovering landmarks, backtracking when necessary, and explicitly declaring when it has reached the destination.
Inferring Maps and Behaviors from Natural Language Instructions
- Computer ScienceISER
- 2014
This paper proposes a probabilistic framework that enables robots to follow commands given in natural language, without any prior knowledge of the environment, and demonstrates the algorithm’s ability to follow navigation commands with performance comparable to that of a fully-known environment.
Learning models for following natural language directions in unknown environments
- Computer Science2015 IEEE International Conference on Robotics and Automation (ICRA)
- 2015
A novel learning framework is proposed that enables robots to successfully follow natural language route directions without any previous knowledge of the environment by learning and performing inference over a latent environment model.
Toward Mobile Robots Reasoning Like Humans
- Computer ScienceAAAI
- 2015
An intelligence architecture that combines cognitive components to carry out high-level cognitive tasks, semantic perception to label regions in the world, and a natural language component to reason about the command and its relationship to the objects in theworld is developed.
Toward Information Theoretic Human-Robot Dialog
- Computer ScienceRobotics: Science and Systems
- 2012
To enable a robot to recover from a failure to understand a natural language utterance, an information-theoretic strategy for asking targeted clarifying questions and using information from the answer to disambiguate the language is described.
Natural language command of an autonomous micro-air vehicle
- Computer Science2010 IEEE/RSJ International Conference on Intelligent Robots and Systems
- 2010
This paper presents a micro-air vehicle capable of following natural language directions through a previously mapped and labeled environment, and demonstrates the robot following directions created by a human for another human, and interactively executing commands in the context of surveillance and search and rescue in confined spaces.
Interactive robot task training through dialog and demonstration
- Computer Science, Psychology2007 2nd ACM/IEEE International Conference on Human-Robot Interaction (HRI)
- 2007
A framework for interactive task training of a mobile robot where the robot learns how to do various tasks while observing a human and how environmental context and communicative dialog with the human help the robot learn the task.
Learning to Parse Natural Language Commands to a Robot Control System
- Computer ScienceISER
- 2012
This work discusses the problem of parsing natural language commands to actions and control structures that can be readily implemented in a robot execution system, and learns a parser based on example pairs of English commands and corresponding control language expressions.
Learning to understand spatial language for robotic navigation and mobile manipulation
- Computer Science
- 2011
A method that uses previously detected objects and places in order to bias the search process toward areas of the environment where a previously unseen object is likely to be found and created on the fly as the robot explores the environment is presented.
A natural language planner interface for mobile manipulators
- Computer Science2014 IEEE International Conference on Robotics and Automation (ICRA)
- 2014
This paper presents a new model called the Distributed Correspondence Graph (DCG) to infer the most likely set of planning constraints from natural language instructions, and presents experimental results from comparative experiments that demonstrate improvements in efficiency in natural language understanding without loss of accuracy.