Visual Dexterity: In-hand Dexterous Manipulation from Depth

@article{Chen2022VisualDI,
  title={Visual Dexterity: In-hand Dexterous Manipulation from Depth},
  author={Tao Chen and Megha Tippur and Siyang Wu and Vikash Kumar and Edward H. Adelson and Pulkit Agrawal},
  journal={ArXiv},
  year={2022},
  volume={abs/2211.11744}
}
In-hand object reorientation is necessary for performing many dexterous manipulation tasks, such as tool use in unstructured environments that remain beyond the reach of current robots. Prior works built reorientation systems that assume one or many of the following specific circumstances: reorienting only specific objects with simple shapes, limited range of reorientation, slow or quasistatic manipulation, the need for specialized and costly sensor suites, simulation-only results, and other… 
View PDF on arXiv
2 Citations
Background Citations
2

Figures and Tables from this paper

2 Citations

Value Guided Exploration with Sub-optimal Controllers for Learning Dexterous Manipulation

This work shows that the framework allows learning from highly sub-optimal controllers and is the first to demonstrate learning hard-to-explore finger-gaiting in-hand manipulation skills without the use of an exploratory reset distribution.

Sampling-based Exploration for Reinforcement Learning of Dexterous Manipulation

This paper presents a novel method for achieving dexterous manipulation of complex objects, while simultaneously securing the object without the use of passive support surfaces using the non-holonomic Rapidly-Exploring Random Trees algorithm.

References

SHOWING 1-10 OF 55 REFERENCES

A System for General In-Hand Object Re-Orientation

This work presents a simple model-free framework that can learn to reorient objects with both the hand facing upwards and downwards and demonstrates the capability of reorienting over 2000 geometrically different objects in both cases.

Transferring Dexterous Manipulation from GPU Simulation to a Remote Real-World TriFinger

This work presents a robot systems approach to learning dexterous manipulation tasks involving moving objects to arbitrary 6-DoF poses, and shows empirical benefits of using keypoint-based representations for object pose in policy observations and reward calculation to train a model-free reinforcement learning agent.

Learning dexterous in-hand manipulation

This work uses reinforcement learning (RL) to learn dexterous in-hand manipulation policies that can perform vision-based object reorientation on a physical Shadow Dexterous Hand, and these policies transfer to the physical robot despite being trained entirely in simulation.

Learning Purely Tactile In-Hand Manipulation with a Torque-Controlled Hand

We show that a purely tactile dextrous in-hand manipulation task with continuous regrasping, requiring permanent force closure, can be learned from scratch and executed robustly on a…

Path Planning for Within-Hand Manipulation over Learned Representations of Safe States

This work proposes a framework for tracking a desired path of an object held by an adaptive hand via within-hand manipulation that can handle objects of different weights without retraining, and successfully avoids undesirable states while minimizing the proposed cost function.

Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations

This work shows that model-free DRL with natural policy gradients can effectively scale up to complex manipulation tasks with a high-dimensional 24-DoF hand, and solve them from scratch in simulated experiments.

Learning Dexterous Manipulation Policies from Experience and Imitation

This work shows that local trajectory-based controllers for complex non-prehensile manipulation tasks can be constructed from surprisingly small amounts of training data, and collections of such controllers can be interpolated to form more global controllers.

Extrinsic dexterity: In-hand manipulation with external forces

This paper studies extrinsic dexterity in the context of regrasp operations, for example when switching from a power to a precision grasp, and demonstrates that even simple grippers are capable of ample in-hand manipulation.

Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost

It is shown that contact-rich manipulation behavior with multi-fingered hands can be learned by directly training with model-free deep RL algorithms in the real world, with minimal additional assumption and without the aid of simulation, indicating that direct deep RL training in thereal world is a viable and practical alternative to simulation and model-based control.

A Dexterous Soft Robotic Hand for Delicate In-Hand Manipulation

The design of a prototype hand with dexterous soft fingers capable of moving objects within the hand using several basic motion primitives is discussed and a simple, heuristic finger gait is examined which enables continuous object rotation for a wide variety of object shapes and sizes.
...