Corpus ID: 235446422

Optical Tactile Sim-to-Real Policy Transfer via Real-to-Sim Tactile Image Translation

  title={Optical Tactile Sim-to-Real Policy Transfer via Real-to-Sim Tactile Image Translation},
  author={Alex Church and J. Lloyd and R. Hadsell and N. Lepora},
Simulation has recently become key for deep reinforcement learning to safely and efficiently acquire general and complex control policies from visual and proprioceptive inputs. Tactile information is not usually considered despite its direct relation to environment interaction. In this work, we present a suite of simulated environments tailored towards tactile robotics and reinforcement learning. A simple and fast method of simulating optical tactile sensors is provided, where high-resolution… Expand
2 Citations
Taxim: An Example-based Simulation Model for GelSight Tactile Sensors
  • Zilin Si, Wenzhen Yuan
  • Computer Science
  • ArXiv
  • 2021
Simulation is widely used in robotics for system verification and large-scale data collection. However, simulating sensors, including tactile sensors, has been a long-standing challenge. In thisExpand
Soft Biomimetic Optical Tactile Sensing with the TacTip: A Review
This article reviews the BRL TacTip as a prototypical example of a SoftBOT (Soft Biomimetic Optical Tactile) sensor and discusses the relation between artificial skin morphology and the transduction principles of human touch. Expand


Sim-to-Real Transfer for Optical Tactile Sensing
A model for soft body simulation which was implemented using the Unity physics engine, and a neural network was trained to predict the locations and angles of edges when in contact with the sensor to accurately predict edges with less than 1 mm prediction error in real-world testing. Expand
Zero-Shot Sim-to-Real Transfer of Tactile Control Policies for Aggressive Swing-Up Manipulation
This letter aims to show that robots equipped with a vision-based tactile sensor can perform dynamic manipulation tasks without prior knowledge of all the physical attributes of the objects to beExpand
TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
TACTO is a fast, flexible and open-source simulator for vision-based tactile sensors that allows to render realistic high-resolution touch readings at hundreds of frames per second, and can be easily configured to simulate different vision- based tactile sensors, including GelSight, DIGIT and OmniTact. Expand
RL-CycleGAN: Reinforcement Learning Aware Simulation-to-Real
The RL-CycleGAN, a new approach for simulation-to-real-world transfer for reinforcement learning, is obtained by incorporating the RL-scene consistency loss into unsupervised domain translation, which ensures that the translation operation is invariant with respect to the Q-values associated with the image. Expand
Tactile Sensing and Deep Reinforcement Learning for In-Hand Manipulation Tasks
Deep Reinforcement Learning techniques demonstrate advances in the domain of robotics. One of the limiting factors is the large number of interaction samples usually required for training inExpand
Sim-to-Real Reinforcement Learning for Deformable Object Manipulation
This work uses a combination of state-of-the-art deep reinforcement learning algorithms to solve the problem of manipulating deformable objects (specifically cloth), and evaluates the approach on three tasks --- folding a towel up to a mark, folding a face towel diagonally, and draping a piece of cloth over a hanger. Expand
Generation of GelSight Tactile Images for Sim2Real Learning
A novel approach for simulating a GelSight tactile sensor in the commonly used Gazebo simulator that can indirectly sense forces, geometry, texture and other properties of the object and enables Sim2Real learning with tactile sensing. Expand
Pose-Based Servo Control with Soft Tactile Sensing
A new way of controlling robots using soft tactile sensors: pose-based tactile servo (PBTS) control is described, to embed a tactile perception model for estimating the sensor pose within a servo control loop that is applied to local object features such as edges and surfaces. Expand
Sim-To-Real via Sim-To-Sim: Data-Efficient Robotic Grasping via Randomized-To-Canonical Adaptation Networks
This paper presents Randomized-to-Canonical Adaptation Networks (RCANs), a novel approach to crossing the visual reality gap that uses no real-world data and learns to translate randomized rendered images into their equivalent non-randomized, canonical versions. Expand
Elastic Interaction of Particles for Robotic Tactile Simulation
Elastic Interaction of Particles (EIP) is proposed, a novel framework for tactile emulation that models the tactile sensor as a group of coordinated particles, and the elastic theory is applied to regulate the deformation of particles during the contact process. Expand