I Know What You Meant: Learning Human Objectives by (Under)estimating Their Choice Set

  title={I Know What You Meant: Learning Human Objectives by (Under)estimating Their Choice Set},
  author={Ananth Jonnavittula and Dylan P. Losey},
  journal={2021 IEEE International Conference on Robotics and Automation (ICRA)},
Assistive robots have the potential to help people perform everyday tasks. However, these robots first need to learn what it is their user wants them to do. Teaching assistive robots is hard for inexperienced users, elderly users, and users living with physical disabilities, since often these individuals are unable to show the robot their desired behavior. We know that inclusive learners should give human teachers credit for what they cannot demonstrate. But today’s robots do the opposite: they… 

Figures from this paper

Theory of Mind and Preference Learning at the Interface of Cognitive Science, Neuroscience, and AI: A Review
This review synthesizes the existing understanding of ToM in cognitive and neurosciences and the AI computational models that have been proposed and focuses on preference learning as an area of particular interest and the most recent neurocognitive and computational ToM models.
Here’s What I’ve Learned: Asking Questions that Reveal Reward Learning
It is found that robots which consider the human’s point of view learn just as quickly as state-of-the-art baselines while also communicating what they have learned to the human operator.
Physical Interaction as Communication: Learning Robot Objectives Online from Human Corrections
This article argues that when pHRI is intentional it is also informative: the robot can leverage interactions to learn how it should complete the rest of its current task even after the person lets go, and improves the efficiency of robot learning from pHRI by reducing unintended learning.


Quantifying Hypothesis Space Misspecification in Learning From Human–Robot Demonstrations and Physical Corrections
It is posited that the robot should reason explicitly about how well it can explain human inputs given its hypothesis space and use that situational confidence to inform how it should incorporate the human input.
Learning from Physical Human Corrections, One Feature at a Time
The approach allows the human-robot team to focus on learning one feature at a time, unlike state-of-the-art techniques that update all features at once, and suggests that users teaching one-at-a-time perform better, especially in tasks that require changing multiple features.
Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences
This work presents an algorithm that first utilizes user demonstrations to initialize a belief about the reward function, and then proactively probes the user with preference queries to zero-in on their true reward, enabling a framework to integrate multiple sources of information, which are either passively or actively collected from human users.
Controlling Assistive Robots with Learned Latent Actions
A teleoperation algorithm for assistive robots that learns latent actions from task demonstrations is designed, and the controllability, consistency, and scaling properties that user-friendly latent actions should have are formulated, and how different lowdimensional embeddings capture these properties are evaluated.
Donut as I do: Learning from failed demonstrations
Instead of maximizing the similarity of generated behaviors to those of the demonstrators, this work examines two methods that deliberately avoid repeating the human's mistakes.
Reward-rational (implicit) choice: A unifying formalism for reward learning
This work provides two examples of how different types of behavior can be interpreted in a single unifying formalism - as a reward-rational choice that the human is making, often implicitly, in the search for a reward.
Active Preference-Based Learning of Reward Functions
This work builds on work in label ranking and proposes to learn from preferences (or comparisons) instead: the person provides the system a relative preference between two trajectories, and takes an active learning approach, in which the system decides on what preference queries to make.
Where Do You Think You're Going?: Inferring Beliefs about Dynamics from Behavior
This paper model suboptimal behavior as the result of internal model misspecification, and demonstrates that this approach enables us to more accurately model human intent, and can be used in a variety of applications, including offering assistance in a shared autonomy framework and inferring human preferences.
Customized Handling of Unintended Interface Operation In Assistive Robots
An assistance system that reasons about a human’s intended actions during robot teleoperation in order to provide appropriate modifications on unintended behavior using model-based inference techniques and the results show that the assistance paradigms helped to significantly reduce task completion time, number of mode switches, cognitive workload, and user frustration, and improve overall user satisfaction.
Choice Set Misspecification in Reward Inference
This work introduces the idea that the choice set itself might be difficult to specify, and analyzes choice set misspecification: what happens as the robot makes incorrect assumptions about the set of choices from which the human selects their feedback.