Policy search for motor primitives in robotics

  title={Policy search for motor primitives in robotics},
  author={Jens Kober and Jan Peters},
  journal={Machine Learning},
Many motor skills in humanoid robotics can be learned using parametrized motor primitives. While successful applications to date have been achieved with imitation learning, most of the interesting motor learning problems are high-dimensional reinforcement learning problems. These problems are often beyond the reach of current reinforcement learning methods. In this paper, we study parametrized policy search methods and apply these to benchmark problems of motor primitive learning in robotics… 
Policy Search for Motor Primitives
An EM-inspired algorithm applicable to complex motor learning tasks is developed and it is shown that it can learn the complex Ball-in-a-Cup task using a real Barrett WAMTM robot arm.
Reinforcement learning of motor skills using Policy Search and human corrective advice
The results show that the proposed method not only converges to higher rewards when learning movement primitives, but also that the learning is sped up by a factor of 4–40 times, depending on the task.
Learning motor skills: from algorithms to robot experiments
  • J. Kober
  • Computer Science
    it Inf. Technol.
  • 2012
It is shown how motor primitives can be employed to learn motor skills on three different levels, which contributes to the state of the art in reinforcement learning applied to robotics both in terms of novel algorithms and applications.
Learning Motor Skills - From Algorithms to Robot Experiments
This book illustrates a method that learns to generalize parameterized motor plans which is obtained by imitation or reinforcement learning, by adapting a small set of global parameters and appropriate kernel-based reinforcement learning algorithms.
Reinforcement Learning to Adjust Robot Movements to New Situations
This paper describes how to learn such mappings from circumstances to meta-parameters using reinforcement learning, and uses a kernelized version of the reward-weighted regression to do so.
Deep Reinforcement Learning for Robotic Manipulation - The state of the art
This work embodies a survey of the most recent algorithms, architectures and their implementations in simulations and real world robotic platforms and manifests some of the state of the art applications of these approaches in robotic manipulation tasks.
Reinforcement learning to adjust parametrized motor primitives to new situations
This paper proposes a method that learns to generalize parametrized motor plans by adapting a small set of global parameters, called meta-parameters, and introduces an appropriate reinforcement learning algorithm based on a kernelized version of the reward-weighted regression.
Deep predictive policy training using reinforcement learning
A data-efficient deep predictive policy training (DPPT) framework with a deep neural network policy architecture which maps an image observation to a sequence of motor activations and is demonstrated by training predictive policies for skilled object grasping and ball throwing on a PR2 robot.
Overcoming Exploration in Reinforcement Learning with Demonstrations
This work uses demonstrations to overcome the exploration problem and successfully learn to perform long-horizon, multi-step robotics tasks with continuous control such as stacking blocks with a robot arm.
Residual Robot Learning for Object-Centric Probabilistic Movement Primitives
This work combines ProMPs with recently introduced Residual Reinforcement Learning (RRL), to account for both, corrections in position and orientation during task execution, and learns a residual on top of a nominal ProMP trajectory with Soft-Actor Critic to reduce the search space for RRL.


Reinforcement Learning for Humanoid Robotics
This paper discusses different approaches of reinforcement learning in terms of their applicability in humanoid robotics, and demonstrates that ‘vanilla’ policy gradient methods can be significantly improved using the natural policy gradient instead of the regular policy gradient.
Policy Gradient Methods for Robotics
  • Jan Peters, S. Schaal
  • Computer Science
    2006 IEEE/RSJ International Conference on Intelligent Robots and Systems
  • 2006
An overview on learning with policy gradient methods for robotics with a strong focus on recent advances in the field is given and how the most recently developed methods can significantly improve learning performance is shown.
Reinforcement learning of motor skills in high dimensions: A path integral approach
This paper derives a novel approach to RL for parameterized control policies based on the framework of stochastic optimal control with path integrals, and believes that this new algorithm, Policy Improvement with Path Integrals (PI2), offers currently one of the most efficient, numerically robust, and easy to implement algorithms for RL in robotics.
Learning motor primitives for robotics
  • J. Kober, Jan Peters
  • Computer Science
    2009 IEEE International Conference on Robotics and Automation
  • 2009
It is shown that two new motor skills, i.e., Ball-in-a-Cup and Ball-Paddling, can be learned on a real Barrett WAM robot arm at a pace similar to human learning while achieving a significantly more reliable final performance.
Learning Attractor Landscapes for Learning Motor Primitives
By nonlinearly transforming the canonical attractor dynamics using techniques from nonparametric regression, almost arbitrary new nonlinear policies can be generated without losing the stability properties of the canonical system.
Learning perceptual coupling for motor primitives
  • J. Kober, B. Mohler, Jan Peters
  • Computer Science, Psychology
    2008 IEEE/RSJ International Conference on Intelligent Robots and Systems
  • 2008
An augmented version of the dynamic system-based motor primitives which incorporates perceptual coupling to an external variable is proposed which can perform complex tasks such a Ball-in-a-Cup or Kendama task even with large variances in the initial conditions where a skilled human player would be challenged.
Reinforcement learning by reward-weighted regression for operational space control
This work uses a generalization of the EM-base reinforcement learning framework suggested by Dayan & Hinton to reduce the problem of learning with immediate rewards to a reward-weighted regression problem with an adaptive, integrated reward transformation for faster convergence.
Machine Learning for motor skills in robotics
This work investigates the ingredients for a general approach to motor skill learning and study two major components for such an approach, i.e., a theoretically well-founded general approach for representing the required control structures for task representation and execution and appropriate learning algorithms which can be applied in this setting.
Towards Direct Policy Search Reinforcement Learning for Robot Control
The policy based algorithm presented in this paper is used for learning the internal state/action mapping of a behavior and is demonstrated with simulated experiments using the underwater robot GARBI in a target reaching task.
Teaching by Showing in Kendama Based on Optimization Principle
A general computational theory is proposed that derives representations for a wide variety of motor behaviors based on the dynamic optimization theory for trajectory formation that is applicable to higher-level task learning.