• Corpus ID: 243832934

LILA: Language-Informed Latent Actions

  title={LILA: Language-Informed Latent Actions},
  author={Siddharth Karamcheti and Megha Srivastava and Percy Liang and Dorsa Sadigh},
: We introduce Language-Informed Latent Actions (LILA), a framework for learning natural language interfaces in the context of human-robot collaboration. LILA falls under the shared autonomy paradigm: in addition to providing discrete language inputs, humans are given a low-dimensional controller – e.g., a 2 degree-of-freedom (DoF) joystick that can move left/right and up/down – for operating the robot. LILA learns to use language to modulate this controller, providing users with a language… 

Shared Autonomy for Robotic Manipulation with Language Corrections

This work presents a method for incorporating language corrections, built on the insight that an initial instruction and subsequent corrections differ mainly in the amount of grounded context needed, to focus on manipulation domains where the sample efficiency of existing work is prohibitive.

Correcting Robot Plans with Natural Language Feedback

This paper describes how to map from natural language sentences to transformations of cost functions and shows that these transformations enable users to correct goals, update robot motions to accommodate additional user preferences, and recover from planning errors.

Towards Learning Generalizable Driving Policies from Restricted Latent Representations

This work capitalize on the key idea that human drivers learn abstract representations of their surroundings that are fairly similar among various driv- ing scenarios and environments to extract a latent representation that minimizes the distance between driving scenarios.



Grounding Language in Play

A simple and scalable way to condition policies on human language instead of language pairing is presented, and a simple technique that transfers knowledge from large unlabeled text corpora to robotic learning is introduced that significantly improves downstream robotic manipulation.

Language-Conditioned Imitation Learning for Robot Manipulation Tasks

This work introduces a method for incorporating unstructured natural language into imitation learning and demonstrates in a set of simulation experiments how this approach can learn language-conditioned manipulation policies for a seven-degree-of-freedom robot arm and compares the results to a variety of alternative methods.

Imitation learning for natural language direction following through unknown environments

This work learns a policy which predicts a sequence of actions that follow the directions by exploring the environment and discovering landmarks, backtracking when necessary, and explicitly declaring when it has reached the destination.

Shared Autonomy with Learned Latent Actions

This work adopts learned latent actions for shared autonomy by proposing a new model structure that changes the meaning of the human's input based on the robot's confidence of the goal, and develops a training procedure to learn a controller that is able to move between goals even in the presence of shared autonomy.

Learning Visually Guided Latent Actions for Assistive Teleoperation

This work develops assistive robots that condition their latent embeddings on visual inputs and indicates that structured visual representations improve few-shot performance and are subjectively preferred by users.

Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text

This work proposes a conceptually simple method for training instruction-following agents with deep RL that are robust to natural human instructions, and demonstrates substantially-above-chance zero-shot transfer from synthetic template commands to natural instructions given by humans.

Controlling Assistive Robots with Learned Latent Actions

A teleoperation algorithm for assistive robots that learns latent actions from task demonstrations is designed, and the controllability, consistency, and scaling properties that user-friendly latent actions should have are formulated, and how different lowdimensional embeddings capture these properties are evaluated.

Grounding English Commands to Reward Functions

This work presents a system that grounds natural language commands into reward functions using demonstrations of differentnatural language commands being carried out in the environment, and demonstrates that the learned model can be both generalized to novel environments and transferred to a robot with a different action space than the action space used during training.

Learning Adaptive Language Interfaces through Decomposition

A neural semantic parsing system that learns new high-level abstractions through decomposition is introduced, demonstrating the flexibility of modern neural systems, as well as the one-shot reliable generalization of grammar-based methods.

Gated-Attention Architectures for Task-Oriented Language Grounding

An end-to-end trainable neural architecture for task-oriented language grounding in 3D environments which assumes no prior linguistic or perceptual knowledge and requires only raw pixels from the environment and the natural language instruction as input.