• Corpus ID: 235313567

Towards Learning to Play Piano with Dexterous Hands and Touch

  title={Towards Learning to Play Piano with Dexterous Hands and Touch},
  author={Huazhe Xu and Yuping Luo and Shaoxiong Wang and Trevor Darrell and Roberto Calandra},
The virtuoso plays the piano with passion, poetry and extraordinary technical ability. As Liszt said “(a virtuoso) must call up scent and blossom, and breathe the breath of life.” The strongest robots that can play a piano are based on a combination of specialized robot hands/piano and hardcoded planning algorithms. In contrast to that, in this paper, we demonstrate how an agent can learn directly from machinereadable music score to play the piano with dexterous hands on a simulated piano using… 

Figures and Tables from this paper

Sim-to-real Deep Reinforcement Learning for Comparing Low-cost High-Resolution Robot Touch
This letter extends the Tactile Gym simulator to include three new optical tactile sensors of the two most popular types, Gelsight- style (image-shading based) and TacTip-style (marker based), and demonstrates that a single sim- to-real approach can be used with these three different sensors to achieve strong real-world performance despite the differences between real tactile images.
Dexterous Manipulation for Multi-Fingered Robotic Hands With Reinforcement Learning: A Review
The purpose is to present a comprehensive review of the techniques for dexterous manipulation with multi-fingered robotic hands, such as the model-based approach without learning in early years, and the latest research and methodologies focused on the method based on reinforcement learning and its variations.


Development of a mini-humanoid pianist
A system that enables a small humanoid to play a music keyboard and also respond to acoustic input and is intended as a proof-of-concept and prototyping platform for humanoid music performance methods, which are ultimately destined for the Hubo adult-sized humanoid.
Piano-Playing Robotic Arm
To fully understand the interpretation and performance of music, the team implemented a set of machine learning techniques, which included training a recurrent neural network (RNN) to analyze audio signals and reproduce musical input.
Learning dexterous in-hand manipulation
This work uses reinforcement learning (RL) to learn dexterous in-hand manipulation policies that can perform vision-based object reorientation on a physical Shadow Dexterous Hand, and these policies transfer to the physical robot despite being trained entirely in simulation.
Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations
This work shows that model-free DRL with natural policy gradients can effectively scale up to complex manipulation tasks with a high-dimensional 24-DoF hand, and solve them from scratch in simulated experiments.
Electronic piano playing robot
An electronic piano playing robot which can automatically play electronic piano according to a user-defined piano sheet music is designed.
More Than a Feeling: Learning to Grasp and Regrasp Using Vision and Touch
An end-to-end action-conditional model that learns regrasping policies from raw visuo-tactile data and outperforms a variety of baselines at estimating grasp adjustment outcomes, selecting efficient grasp adjustments for quick grasping, and reducing the amount of force applied at the fingers, while maintaining competitive performance.
Piano playing robot
The aim of this project is to build a robot which can play piano which uses a simple DC motor which pulls the finger down and thus presses a key on the keyboard, thus enabling the robot to play various tunes.
Force control for the fingers of the piano playing robot — A gain switched approach
In this paper, a force controller design is considered for the fingers of the piano playing robot with a gain switched controller. Music playing robot is one kind of the service robots. In addition
TACTO: A Fast, Flexible, and Open-Source Simulator for High-Resolution Vision-Based Tactile Sensors
TACTO is a step towards the widespread adoption of touch sensing in robotic applications, and to enable machine learning practitioners interested in multi-modal learning and control, and is provided a proof-of-concept that TACTO can be successfully used for Sim2Real applications.
SwingBot: Learning Physical Features from In-hand Tactile Exploration for Dynamic Swing-up Manipulation
S SwingBot is a robot that is able to learn the physical features of an held object through tactile exploration, and with the learned physical features its end-to-end self-supervised learning pipeline is ability to substantially improve the accuracy of swinging up unseen objects.