DEP-RL: Embodied Exploration for Reinforcement Learning in Overactuated and Musculoskeletal Systems

  title={DEP-RL: Embodied Exploration for Reinforcement Learning in Overactuated and Musculoskeletal Systems},
  author={Pierre Schumacher and Daniel F. B. Haeufle and Dieter B{\"u}chler and Syn Schmitt and Georg Martius},
Muscle-actuated organisms are capable of learning an unparalleled diversity of dexterous movements despite their vast amount of muscles. Reinforcement learning (RL) on large musculoskeletal models, however, has not been able to show similar performance. We conjecture that ineffective exploration in large overactuated action spaces is a key problem. This is supported by the finding that common exploration noise strategies are inadequate in synthetic examples of overactuated systems. We identify… 

Learning with Muscles: Benefits for Data-Efficiency and Robustness in Anthropomorphic Tasks

This work hypothesizes that highly nonlinear muscle dynamics play a large role in providing inherent stability, which is favorable to learning, and showcases the potential of muscle actuators for core robotics challenges in terms of data-efficiency, hyperparameter sensitivity, and robustness.



Reaching control of a full-torso, modelled musculoskeletal robot using muscle synergies emergent under reinforcement learning

A novel controller is derived for this class of robot which learns effective reaching actions through the sustained activation of weighted muscle synergies, an approach which draws upon compelling, recent evidence from animal and human studies, but is almost unexplored to date in the musculoskeletal robot literature.

Learning to Control Redundant Musculoskeletal Systems with Neural Networks and SQP: Exploiting Muscle Properties

The approach uses a learned forward model represented by a neural network and sequential quadratic programming to obtain the control policy, which allows to change the stiffness of the system and to include optimality criteria like small muscle stimulations.

Reinforcement learning control of a biomechanical model of the upper extremity

The results confirm that the assumptions of signal-dependent and constant motor noise, together with the objective of movement time minimization, are sufficient for a state-of-the-art skeletal model of the human upper extremity to reproduce complex phenomena of human movement, in particular Fitts’ Law and the Power Law.

Reinforcement Learning of Musculoskeletal Control from Functional Simulations

In this work, a deep reinforcement learning (DRL) based inverse dynamics controller is trained to control muscle activations of a biomechanical model of the human shoulder using the evolving neural models of the DRL being trained.

The effects of motor modularity on performance, learning and generalizability in upper-extremity reaching: a computational analysis

It is found that a controller based on synergies had errors within 1 cm of a full-dimensional controller and achieved faster learning rates (as estimated from computational time to converge), and the synergy-based controllers could also accomplish new tasks—such as reaching to targets on a higher or lower plane, and starting from alternative initial poses—with average errors similar to a full -dimensional controller.

Hindsight Experience Replay Improves Reinforcement Learning for Control of a MIMO Musculoskeletal Model of the Human Arm

The ability of a controller trained using reinforcement learning to generate desired movements of a horizontal planar musculoskeletal model of the human arm with 2 degrees of freedom and 6 actuators is demonstrated.

OstrichRL: A Musculoskeletal Ostrich Simulation to Study Bio-mechanical Locomotion

A 3D musculoskeletal simulation of an ostrich based on the MuJoCo simulator that can be a useful bridge between the biomechanics, reinforcement learning, graphics and robotics communities, by providing a fast and easy to use simulation.

Learning Continuous Muscle Control for a Multi-joint Arm by Extending Proximal Policy Optimization with a Liquid State Machine

This work uses reinforcement learning for a target reaching task, which can be modeled as partially observable markov decision processes, and shows that there are clear benefits of training the readouts of a liquid state machine (LSM) with reinforcement learning.

Robust Walking Control of a Lower Limb Rehabilitation Exoskeleton Coupled with a Musculoskeletal Model via Deep Reinforcement Learning

Background: Few studies have systematically investigated robust controllers for lower limb rehabilitation exoskeletons (LLREs) that can safely and effectively assist users with a variety of

A lightweight robotic arm with pneumatic muscles for robot learning

A light-weight robot arm with moving masses below 700 g with powerful antagonistic compliant actuation with pneumatic artificial muscles that will enable new applications in fast changing and uncertain task like robot table tennis while being a sophisticated and reproducible test-bed for robot skill learning methods.