Generalization through Simulation: Integrating Simulated and Real Data into Deep Reinforcement Learning for Vision-Based Autonomous Flight
@article{Kang2019GeneralizationTS, title={Generalization through Simulation: Integrating Simulated and Real Data into Deep Reinforcement Learning for Vision-Based Autonomous Flight}, author={Katie Kang and Suneel Belkhale and Gregory Kahn and P. Abbeel and Sergey Levine}, journal={2019 International Conference on Robotics and Automation (ICRA)}, year={2019}, pages={6008-6014} }
Deep reinforcement learning provides a promising approach for vision-based control of real-world robots. However, the generalization of such models depends critically on the quantity and variety of data available for training. This data can be difficult to obtain for some types of robotic systems, such as fragile, small-scale quadrotors. Simulated rendering and physics can provide for much larger datasets, but such data is inherently of lower quality: many of the phenomena that make the real…
82 Citations
Toward Real-World Implementation of Deep Reinforcement Learning for Vision-Based Autonomous Drone Navigation with Mission
- Computer Science
- 2022
This work focuses on using object detectors that tend to transfer well from simulation to the real world, and extract features of detected objects to serve as input to reinforcement learning algorithms.
Stronger Generalization Guarantees for Robot Learning by Combining Generative Models and Real-World Data
- Computer Science2022 International Conference on Robotics and Automation (ICRA)
- 2022
The key idea behind the approach is to utilize the generative model in order to implicitly specify a prior over policies, updated using the real-world dataset of environments by minimizing an upper bound on the expected cost across novel environments derived via Probably Approximately Correct (PAC)-Bayes generalization theory.
Simulation-Based Reinforcement Learning for Real-World Autonomous Driving
- Computer Science2020 IEEE International Conference on Robotics and Automation (ICRA)
- 2020
This work uses reinforcement learning in simulation to obtain a driving system controlling a full-size real-world vehicle that takes RGB images from a single camera and their semantic segmentation as input and achieves successful sim-to-real policy transfer.
Domain Generalization via Optical Flow: Training a CNN in a Low-Quality Simulation to Detect Obstacles in the Real World
- Computer Science2020 17th Conference on Computer and Robot Vision (CRV)
- 2020
This work trains a neural network to detect collisions with simulated optical flow data and easily achieves a higher detection accuracy when compared to a network trained on a similar dataset of realworld collisions.
Vision-Based Multirotor Following Using Synthetic Learning Techniques
- Computer ScienceSensors
- 2019
Results confirm that the present framework can be used to deploy a vision-based task in real flight using synthetic data and introduce a novel motion-control strategy for object following where the camera gimbal movement is coupled with themultirotor motion during the multirotor following.
Learning Visuomotor Policies for Aerial Navigation Using Cross-Modal Representations
- Computer Science2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
- 2020
This work proposes a novel method for learning robust visuomotor policies for real-world deployment which can be trained purely with simulated data, and develops rich state representations that combine supervised and unsupervised environment data.
Deep Reinforcement Learning for Autonomous Map-less Navigation of a Flying Robot
- Computer ScienceIEEE Access
- 2022
A novel approach for enabling a Micro Aerial Vehicle system equipped with a laser rangefinder and depth sensor to autonomously navigate and explore an unknown indoor or outdoor environment by building a modular Deep-Q-Network architecture to fuse information from multiple sensors mounted onboard the vehicle.
Sim-to-Real Deep Reinforcement Learning for Safe End-to-End Planning of Aerial Robots
- Computer ScienceRobotics
- 2022
A novel end-to-end path planning algorithm based on deep reinforcement learning is proposed for aerial robots deployed in dense environments that finds an obstacle-free way around the provided rough, global path by only depending on the observations from a forward-facing depth camera.
Combining Optimal Control and Learning for Visual Navigation in Novel Environments
- Computer ScienceCoRL
- 2019
This work coupling model-based control with learning-based perception produces a series of waypoints that guide the robot to the goal via a collision-free path and demonstrates that the proposed approach can reach goal locations more reliably and efficiently in novel environments as compared to purely geometric mapping-based or end-to-end learning- based alternatives.
Crossing the Reality Gap: a Survey on Sim-to-Real Transferability of Robot Controllers in Reinforcement Learning
- Computer ScienceIEEE Access
- 2021
The goal of the present survey is to identify the main approaches to face the RG problem in the context of robot control with RL, to point out their shortcomings, and to outline new potential research areas.
References
SHOWING 1-10 OF 46 REFERENCES
Sim-to-Real Robot Learning from Pixels with Progressive Nets
- Computer ScienceCoRL
- 2017
This work proposes using progressive networks to bridge the reality gap and transfer learned policies from simulation to the real world, and presents an early demonstration of this approach with a number of experiments in the domain of robot manipulation that focus on bridging thereality gap.
Reinforcement learning for non-prehensile manipulation: Transfer from simulation to physical system
- Computer Science2018 IEEE International Conference on Simulation, Modeling, and Programming for Autonomous Robots (SIMPAR)
- 2018
It is shown that control policies learned in simulation can successfully transfer to a physical system, composed of three Phantom robots pushing an object to various desired target positions, using a modified form of the natural policy gradient algorithm applied to a carefully identified simulation model.
Asymmetric Actor Critic for Image-Based Robot Learning
- Computer ScienceRobotics: Science and Systems
- 2018
This work exploits the full state observability in the simulator to train better policies which take as input only partial observations (RGBD images) and combines this method with domain randomization and shows real robot experiments for several tasks like picking, pushing, and moving a block.
Sim-to-Real Transfer of Robotic Control with Dynamics Randomization
- Computer Science2018 IEEE International Conference on Robotics and Automation (ICRA)
- 2018
By randomizing the dynamics of the simulator during training, this paper is able to develop policies that are capable of adapting to very different dynamics, including ones that differ significantly from the dynamics on which the policies were trained.
Modular Deep Q Networks for Sim-to-real Transfer of Visuo-motor Policies
- Computer ScienceICRA 2017
- 2016
A modular deep reinforcement learning method capable of transferring models trained in simulation to a real-world robotic task, and introduces a bottleneck between perception and control, enabling the networks to be trained independently, but then merged and fine-tuned in an end-to-end manner to further improve hand-eye coordination.
(CAD)$^2$RL: Real Single-Image Flight without a Single Real Image
- Computer ScienceRobotics: Science and Systems
- 2017
This paper proposes a learning method that they call CAD$^2$RL, which can be used to perform collision-free indoor flight in the real world while being trained entirely on 3D CAD models, and shows that it can train a policy that generalizes to thereal world, without requiring the simulator to be particularly realistic or high-fidelity.
Self-Supervised Deep Reinforcement Learning with Generalized Computation Graphs for Robot Navigation
- Computer Science2018 IEEE International Conference on Robotics and Automation (ICRA)
- 2018
A generalized computation graph is proposed that subsumes value-based model-free methods and model-based methods, and is instantiate to form a navigation model that learns from raw images and is sample efficient, and outperforms single-step and double-step double Q-learning.
Sample-efficient Reinforcement Learning via Difference Models
- Computer Science
- 2018
This paper develops an iterative model learning approach which learns the simulation gap and provides two benefits: it can scale to systems with highly non-linear and contactrich dynamics with continuous state and action spaces and provide sample complexity results.
VR-Goggles for Robots: Real-to-Sim Domain Adaptation for Visual Control
- Computer ScienceIEEE Robotics and Automation Letters
- 2019
This letter proposes a simple yet effective shift loss that is agnostic to the downstream task, to constrain the consistency between subsequent frames which is important for consistent policy outputs, and validate the shift loss for artistic style transfer for videos and domain adaptation.
Reinforcement learning in robotics: A survey
- Computer ScienceInt. J. Robotics Res.
- 2012
This article attempts to strengthen the links between the two research communities by providing a survey of work in reinforcement learning for behavior generation in robots by highlighting both key challenges in robot reinforcement learning as well as notable successes.