Generalization through Simulation: Integrating Simulated and Real Data into Deep Reinforcement Learning for Vision-Based Autonomous Flight

  title={Generalization through Simulation: Integrating Simulated and Real Data into Deep Reinforcement Learning for Vision-Based Autonomous Flight},
  author={Katie Kang and Suneel Belkhale and Gregory Kahn and P. Abbeel and Sergey Levine},
  journal={2019 International Conference on Robotics and Automation (ICRA)},
Deep reinforcement learning provides a promising approach for vision-based control of real-world robots. However, the generalization of such models depends critically on the quantity and variety of data available for training. This data can be difficult to obtain for some types of robotic systems, such as fragile, small-scale quadrotors. Simulated rendering and physics can provide for much larger datasets, but such data is inherently of lower quality: many of the phenomena that make the real… 

Figures and Tables from this paper

Toward Real-World Implementation of Deep Reinforcement Learning for Vision-Based Autonomous Drone Navigation with Mission

This work focuses on using object detectors that tend to transfer well from simulation to the real world, and extract features of detected objects to serve as input to reinforcement learning algorithms.

Stronger Generalization Guarantees for Robot Learning by Combining Generative Models and Real-World Data

The key idea behind the approach is to utilize the generative model in order to implicitly specify a prior over policies, updated using the real-world dataset of environments by minimizing an upper bound on the expected cost across novel environments derived via Probably Approximately Correct (PAC)-Bayes generalization theory.

Simulation-Based Reinforcement Learning for Real-World Autonomous Driving

This work uses reinforcement learning in simulation to obtain a driving system controlling a full-size real-world vehicle that takes RGB images from a single camera and their semantic segmentation as input and achieves successful sim-to-real policy transfer.

Domain Generalization via Optical Flow: Training a CNN in a Low-Quality Simulation to Detect Obstacles in the Real World

This work trains a neural network to detect collisions with simulated optical flow data and easily achieves a higher detection accuracy when compared to a network trained on a similar dataset of realworld collisions.

Vision-Based Multirotor Following Using Synthetic Learning Techniques

Results confirm that the present framework can be used to deploy a vision-based task in real flight using synthetic data and introduce a novel motion-control strategy for object following where the camera gimbal movement is coupled with themultirotor motion during the multirotor following.

Learning Visuomotor Policies for Aerial Navigation Using Cross-Modal Representations

This work proposes a novel method for learning robust visuomotor policies for real-world deployment which can be trained purely with simulated data, and develops rich state representations that combine supervised and unsupervised environment data.

Deep Reinforcement Learning for Autonomous Map-less Navigation of a Flying Robot

A novel approach for enabling a Micro Aerial Vehicle system equipped with a laser rangefinder and depth sensor to autonomously navigate and explore an unknown indoor or outdoor environment by building a modular Deep-Q-Network architecture to fuse information from multiple sensors mounted onboard the vehicle.

Sim-to-Real Deep Reinforcement Learning for Safe End-to-End Planning of Aerial Robots

A novel end-to-end path planning algorithm based on deep reinforcement learning is proposed for aerial robots deployed in dense environments that finds an obstacle-free way around the provided rough, global path by only depending on the observations from a forward-facing depth camera.

Combining Optimal Control and Learning for Visual Navigation in Novel Environments

This work coupling model-based control with learning-based perception produces a series of waypoints that guide the robot to the goal via a collision-free path and demonstrates that the proposed approach can reach goal locations more reliably and efficiently in novel environments as compared to purely geometric mapping-based or end-to-end learning- based alternatives.

Crossing the Reality Gap: a Survey on Sim-to-Real Transferability of Robot Controllers in Reinforcement Learning

The goal of the present survey is to identify the main approaches to face the RG problem in the context of robot control with RL, to point out their shortcomings, and to outline new potential research areas.



Sim-to-Real Robot Learning from Pixels with Progressive Nets

This work proposes using progressive networks to bridge the reality gap and transfer learned policies from simulation to the real world, and presents an early demonstration of this approach with a number of experiments in the domain of robot manipulation that focus on bridging thereality gap.

Reinforcement learning for non-prehensile manipulation: Transfer from simulation to physical system

It is shown that control policies learned in simulation can successfully transfer to a physical system, composed of three Phantom robots pushing an object to various desired target positions, using a modified form of the natural policy gradient algorithm applied to a carefully identified simulation model.

Asymmetric Actor Critic for Image-Based Robot Learning

This work exploits the full state observability in the simulator to train better policies which take as input only partial observations (RGBD images) and combines this method with domain randomization and shows real robot experiments for several tasks like picking, pushing, and moving a block.

Sim-to-Real Transfer of Robotic Control with Dynamics Randomization

By randomizing the dynamics of the simulator during training, this paper is able to develop policies that are capable of adapting to very different dynamics, including ones that differ significantly from the dynamics on which the policies were trained.

Modular Deep Q Networks for Sim-to-real Transfer of Visuo-motor Policies

A modular deep reinforcement learning method capable of transferring models trained in simulation to a real-world robotic task, and introduces a bottleneck between perception and control, enabling the networks to be trained independently, but then merged and fine-tuned in an end-to-end manner to further improve hand-eye coordination.

(CAD)$^2$RL: Real Single-Image Flight without a Single Real Image

This paper proposes a learning method that they call CAD$^2$RL, which can be used to perform collision-free indoor flight in the real world while being trained entirely on 3D CAD models, and shows that it can train a policy that generalizes to thereal world, without requiring the simulator to be particularly realistic or high-fidelity.

Self-Supervised Deep Reinforcement Learning with Generalized Computation Graphs for Robot Navigation

A generalized computation graph is proposed that subsumes value-based model-free methods and model-based methods, and is instantiate to form a navigation model that learns from raw images and is sample efficient, and outperforms single-step and double-step double Q-learning.

Sample-efficient Reinforcement Learning via Difference Models

This paper develops an iterative model learning approach which learns the simulation gap and provides two benefits: it can scale to systems with highly non-linear and contactrich dynamics with continuous state and action spaces and provide sample complexity results.

VR-Goggles for Robots: Real-to-Sim Domain Adaptation for Visual Control

This letter proposes a simple yet effective shift loss that is agnostic to the downstream task, to constrain the consistency between subsequent frames which is important for consistent policy outputs, and validate the shift loss for artistic style transfer for videos and domain adaptation.

Reinforcement learning in robotics: A survey

This article attempts to strengthen the links between the two research communities by providing a survey of work in reinforcement learning for behavior generation in robots by highlighting both key challenges in robot reinforcement learning as well as notable successes.