Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection

@article{Levine2018LearningHC,
  title={Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection},
  author={Sergey Levine and Peter Pastor and Alex Krizhevsky and Deirdre Quillen},
  journal={The International Journal of Robotics Research},
  year={2018},
  volume={37},
  pages={421 - 436}
}
We describe a learning-based approach to hand-eye coordination for robotic grasping from monocular images. To learn hand-eye coordination for grasping, we trained a large convolutional neural network to predict the probability that task-space motion of the gripper will result in successful grasps, using only monocular camera images independent of camera calibration or the current robot pose. This requires the network to observe the spatial relationship between the gripper and objects in the… 

Figures and Tables from this paper

Learning Hand-Eye Coordination for Robotic Grasping with Large-Scale Data Collection
TLDR
A large convolutional neural network is trained to predict the probability that task-space motion of the gripper will result in successful grasps, using only monocular camera images and independently of camera calibration or the current robot pose.
Learning from Successes and Failures to Grasp Objects with a Vacuum Gripper
TLDR
This work employs a Convolutional Neural Network that directly infers the grasping points and the approach angles from RGB-D images as a regression problem, and employs a self-supervised, data-driven learning approach to estimate a suitable grasp for known and unknown objects.
Learning a visuomotor controller for real world robotic grasping using easily simulated depth images
TLDR
This paper proposes an approach to learning a closed-loop controller for robotic grasping that dynamically guides the gripper to the object and significantly outperforms the baseline in the presence of kinematic noise, perceptual errors and disturbances of the object during grasping.
Learning a visuomotor controller for real world robotic grasping using simulated depth images
TLDR
This paper proposes an approach to learning a closed-loop controller for robotic grasping that dynamically guides the gripper to the object and finds that this approach significantly outperforms the baseline in the presence of kinematic noise, perceptual errors and disturbances of the object during grasping.
Learning Object Grasping for Soft Robot Hands
TLDR
The power of a 3D CNN model is exploited to estimate suitable grasp poses from multiple grasping directions (top and side directions) and wrist orientations, which has great potential for geometry-related robotic tasks.
Fast Convolutional Neural Network for Real-Time Robotic Grasp Detection
TLDR
This work addresses the visual perception phase involved in the robotic grasping problem by obtaining a grasping rectangle that symbolizes the position, orientation, and opening of the robot's parallel grippers just before the grippers are closed.
Robotic Grasping using Demonstration and Deep Learning
TLDR
A data collection system that can be used to collect a large number of human demonstrated grasps and identifies a strong prior in the dataset over the distribution of grasp positions and leverage this information by tasking the position network to predict corrections to this prior based on the image being presented to it.
Grasp for Stacking via Deep Reinforcement Learning
TLDR
A model-free deep Q-learning method to learn the grasping-stacking strategy end-to-end from scratch through two deep networks, which maps the images to the actions of the robotic arm throughTwo deep networks: the grasping network (GNet) using the observation of the desk and the pile to infer the gripper’s position and orientation for grasping, and the stacking network (SNet).
RPRG: Toward Real-time Robotic Perception, Reasoning and Grasping with One Multi-task Convolutional Neural Network
TLDR
A multi-task convolutional neural network is proposed for Robotic Perception, Reasoning and Grasping (RPRG), which can help robot find the target, make the plan for grasping and finally grasp the target step by step in object stacking scenes.
DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Video
TLDR
Through experiments on 27 objects with a 30-DoF simulated robot hand, it is demonstrated that DexVIP compares favorably to existing approaches that lack a hand pose prior or rely on specialized tele-operation equipment to obtain human demonstrations, while also being faster to train.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 68 REFERENCES
Learning Hand-Eye Coordination for Robotic Grasping with Large-Scale Data Collection
TLDR
A large convolutional neural network is trained to predict the probability that task-space motion of the gripper will result in successful grasps, using only monocular camera images and independently of camera calibration or the current robot pose.
Deep learning for detecting robotic grasps
TLDR
This work presents a two-step cascaded system with two deep networks, where the top detections from the first are re-evaluated by the second, and shows that this method improves performance on an RGBD robotic grasping dataset, and can be used to successfully execute grasps on two different robotic platforms.
Generalization of human grasping for multi-fingered robot hands
TLDR
An imitation learning approach for learning and generalizing grasping skills based on human demonstrations is presented, which learns low-dimensional latent grasp spaces for different grasp types which form the basis for a novel extension to dynamic motor primitives.
Using Near-Field Stereo Vision for Robotic Grasping in Cluttered Environments
TLDR
A simple but robust approach to both pre-touch grasp adjustment and grasp planning for unknown objects in clutter, using a small-baseline stereo camera attached to the gripper of the robot and a feature-based cost function on local 3D data.
Active vision for dexterous grasping of novel objects
TLDR
This work answers the question of how should a robot direct active vision so as to ensure reliable grasping for dexterous grasping of unfamiliar objects and shows that this approach outperforms a randomised algorithm.
Deep learning a grasp function for grasping under gripper pose uncertainty
TLDR
A new method for parallel-jaw grasping of isolated objects from depth images, under large gripper pose uncertainty, which trains a Convolutional Neural Network which takes as input a single depth image of an object, and outputs a score for each grasp pose across the image.
Visual servoing for humanoid grasping and manipulation tasks
TLDR
It is shown how a robust visual perception is used to control complex robots without any hand-eye calibration and the robustness of the system is improved by estimating the hand position in case of failed visual hand tracking due to lightning artifacts or occlusions.
Supersizing self-supervision: Learning to grasp from 50K tries and 700 robot hours
  • Lerrel Pinto, A. Gupta
  • Computer Science
    2016 IEEE International Conference on Robotics and Automation (ICRA)
  • 2016
TLDR
This paper takes the leap of increasing the available training data to 40 times more than prior work, leading to a dataset size of 50K data points collected over 700 hours of robot grasping attempts, which allows us to train a Convolutional Neural Network for the task of predicting grasp locations without severe overfitting.
Focused online visual-motor coordination for a dual-arm robot manipulator
TLDR
A novel visual-motor coordination method that performs online parameter estimation of an RGB-D camera mounted in a robot head without any external markers that outperforms state-of-the-art offline registration methods in terms of accuracy and computation time.
Robot arm pose estimation by pixel-wise regression of joint angles
TLDR
This work proposes an approach for robot arm pose estimation that uses depth images of the arm as input to directly estimate angular joint positions and shows that this approach improves previous work both in terms of computational complexity and accuracy.
...
1
2
3
4
5
...