• Corpus ID: 224725337

Model-Based Inverse Reinforcement Learning from Visual Demonstrations

@inproceedings{Das2020ModelBasedIR,
  title={Model-Based Inverse Reinforcement Learning from Visual Demonstrations},
  author={Neha Das and Sarah Bechtle and Todor Davchev and Dinesh Jayaraman and Akshara Rai and Franziska Meier},
  booktitle={CoRL},
  year={2020}
}
Scaling model-based inverse reinforcement learning (IRL) to real robotic manipulation tasks with unknown dynamics remains an open problem. The key challenges lie in learning good dynamics models, developing algorithms that scale to high-dimensional state-spaces and being able to learn from both visual and proprioceptive demonstrations. In this work, we present a gradient-based inverse reinforcement learning framework that utilizes a pre-trained visual dynamics model to learn cost functions when… 

Figures and Tables from this paper

Learning Extended Body Schemas from Visual Keypoints for Object Manipulation
TLDR
This work develops an approach that can extend a robot's kinematic model when grasping an object from visual latent representations, and learns to consistently predict visual keypoints on objects, and can easily adapt a kinematics chain to the object grasped in various configurations.
Learning Time-Invariant Reward Functions through Model-Based Inverse Reinforcement Learning
TLDR
This work proposes a formulation that allows to vary the length of execution by learning time-invariant costs, and relax the temporal alignment requirements for learning from demonstration and shows that this approach enables learning temporally invariant rewards from misaligned demonstration that can also generalise spatially to out of distribution tasks.
Multi-Modal Learning of Keypoint Predictive Models for Visual Object Manipulation
TLDR
This work develops an selfsupervised approach that can extend a robot’s kinematic model when grasping an object from visual latent representations, and learns to consistently predict visual keypoints on objects in the manipulator's hand.
Modeling Human Behavior Part I - Learning and Belief Approaches
TLDR
The main objective of this paper is to provide a succinct yet systematic review of the most important approaches in two areas dealing with quantitative models of human behaviors, and to directly model mechanisms of human reasoning, such as beliefs and bias, without going necessarily learning via trial-and-error.
R3M: A Universal Visual Representation for Robot Manipulation
TLDR
This work pre-train a visual representation using the Ego4D human video dataset using a combination of time-contrastive learning, video-language alignment, and an L1 penalty to encourage sparse and compact representations, resulting in R3M.
Vision-based system identification and 3D keypoint discovery using dynamics constraints
TLDR
A novel method that enables simultaneous keypoint discovery, 3D system identification, and extrinsic camera calibration from an unlabeled video taken from a static camera, using only the family of equations of motion of the object of interest as weak supervision is introduced.
A Differentiable Recipe for Learning Visual Non-Prehensile Planar Manipulation
TLDR
This work proposes a novel architecture, Differentiable Learning for Manipulation (DLM), that combines video decoding neural models with priors from contact mechanics by leveraging differentiable optimization and finite difference based simulation.
Concept2Robot: Learning manipulation concepts from instructions and human demonstrations
TLDR
This work aims to endow a robot with the ability to learn manipulation concepts that link natural language instructions to motor skills by proposing a two-stage learning process where the robot first learns single-task policies through reinforcement learning and a multi-task policy through imitation learning.
Back to Reality for Imitation Learning
TLDR
This paper proposes that the most appropriate evaluation metric for robot learning is not data efficiency, but time efficiency, which captures the real-world cost much more truthfully.
A Differentiable Recipe for Learning Visual Dexterous Planar Manipulation
  • Computer Science
  • 2021
TLDR
This work proposes a novel learning architecture that combines video decoding neural models with priors from contact mechanics by leveraging differentiable optimization and differentiable simulation and finds that its modular and fully differentiable architecture outperforms learning-only methods on unseen objects and motions.
...
...

References

SHOWING 1-10 OF 34 REFERENCES
Approximate Inverse Reinforcement Learning from Vision-based Imitation Learning
TLDR
The method uses Imitation Learning as a means to do Inverse Reinforcement Learning in order to create an approximate cost function generator for a visual navigation challenge, which outperforms other state-of-the-art costmap generators in novel environments.
Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
TLDR
It is demonstrated that visual MPC can generalize to never-before-seen objects---both rigid and deformable---and solve a range of user-defined object manipulation tasks using the same model.
Learning objective functions for manipulation
TLDR
An approach to learning objective functions for robotic manipulation based on inverse reinforcement learning that can deal with high-dimensional continuous state-action spaces, and only requires local optimality of demonstrated trajectories is presented.
Deep visual foresight for planning robot motion
  • Chelsea Finn, S. Levine
  • Computer Science
    2017 IEEE International Conference on Robotics and Automation (ICRA)
  • 2017
TLDR
This work develops a method for combining deep action-conditioned video prediction models with model-predictive control that uses entirely unlabeled training data and enables a real robot to perform nonprehensile manipulation — pushing objects — and can handle novel objects not seen during training.
Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
TLDR
This work explores how inverse optimal control (IOC) can be used to learn behaviors from demonstrations, with applications to torque control of high-dimensional robotic systems and an efficient sample-based approximation for MaxEnt IOC.
Unsupervised Perceptual Rewards for Imitation Learning
TLDR
This work presents a method that is able to identify key intermediate steps of a task from only a handful of demonstration sequences, and automatically identify the most discriminative features for identifying these steps.
Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation
TLDR
This work proposes an imitation learning method based on video prediction with context translation and deep reinforcement learning that enables a variety of interesting applications, including learning robotic skills that involve tool use simply by observing videos of human tool use.
Inverse Reinforcement Learning with Model Predictive Control
TLDR
This article introduces a novel scheme of inverse reinforcement learning to take advantage of the feature identification capability of neural network to determine the reward function of model predictive control in order to solve the practical autonomous driving longitudinal control problem.
SE3-Pose-Nets: Structured Deep Dynamics Models for Visuomotor Control
TLDR
This model, a variant of SE3-Nets, learns a low-dimensional pose embedding for visuomotor control via an encoder-decoder structure and is robust, runs in real-time, achieves good prediction of scene dynamics, and outperforms baselines on multiple control runs.
Maximum Entropy Inverse Reinforcement Learning
TLDR
A probabilistic approach based on the principle of maximum entropy that provides a well-defined, globally normalized distribution over decision sequences, while providing the same performance guarantees as existing methods is developed.
...
...