• Corpus ID: 214714285

Learning to Smooth and Fold Real Fabric Using Dense Object Descriptors Trained on Synthetic Color Images

  title={Learning to Smooth and Fold Real Fabric Using Dense Object Descriptors Trained on Synthetic Color Images},
  author={Aditya Ganapathi and Priya Sundaresan and Brijen Thananjeyan and Ashwin Balakrishna and Daniel Seita and Jennifer Grannen and Minho Hwang and Ryan Hoque and Joseph Gonzalez and Nawid Jamali and Katsu Yamane and Soshi Iba and Ken Goldberg},
Robotic fabric manipulation is challenging due to the infinite dimensional configuration space and complex dynamics. In this paper, we learn visual representations of deformable fabric by training dense object descriptors that capture correspondences across images of fabric in various configurations. The learned descriptors capture higher level geometric structure, facilitating design of explainable policies. We demonstrate that the learned representation facilitates multistep fabric smoothing… 

Learning Arbitrary-Goal Fabric Folding with One Hour of Real Robot Experience

This paper shows that it is possible to learn fabric folding skills in only an hour of self-supervised real robot experience, without human supervision or simulation, and creates an expressive goal-conditioned pick and place policy that can be trained efficiently with real world robot data only.

MMGSD: Multi-Modal Gaussian Shape Descriptors for Correspondence Matching in 1D and 2D Deformable Objects

Multi-Modal Gaussian Shape Descriptor (MMGSD), a new visual representation of deformable objects which extends ideas from dense object descriptors to predict all symmetric correspondences between different object configurations, is proposed.

Generalizing Object-Centric Task-Axes Controllers using Keypoints

This work proposes to learn modular task policies which compose object-centric task-axes controllers which are parameterized by properties associated with underlying objects in the scene and infer these controller parameters directly from visual input using multi-view dense correspondence learning.

Untangling Dense Knots by Learning Task-Relevant Keypoints

An algorithm, HULK: Hierarchical Untangling from Learned Keypoints, which combines learning-based perception with a geometric planner into a policy that guides a bilateral robot to untangle knots isInstantiated into an algorithm that is able to untangled cables with dense figure-eight and overhand knots and generalize to varied textures and appearances.

Untangling Dense Non-Planar Knots by Learning Manipulation Features and Recovery Policies

The combination of HULK, LOKI, and SPiDERMan is able to untangle dense overhand, figure-eight, double-overhand, square, bowline, granny, stevedore, and triple-over hand knots.

NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fields

This paper demonstrates that a NeRF representation of a scene can be used to train dense object descriptors and demonstrates the learned dense descriptors enable robots to perform accurate 6-degree of freedom (6-DoF) pick and place of thin and reflective objects.

Learning Robot Policies for Untangling Dense Knots in Linear Deformable Structures

HULK is able to successfully untangle an LDO from a dense initial configuration containing only up to two overhand and figure-eight knots in 97.9% of 378 simulated experiments with an average of 12.1 actions per trial, suggesting that the policy can learn the task of untangling effectively from an algorithmic supervisor.

RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks

This work shows through experiments that with just 10 minutes of real-world robotic interaction data, the RoboCraft robot can learn a dynamics model that can be used to synthesize control signals to deform elasto-plastic objects into various target shapes, including shapes that the robot has never encountered before.

Deformable Elasto-Plastic Object Shaping using an Elastic Hand and Model-Based Reinforcement Learning

  • Carolyn MatlR. Bajcsy
  • Computer Science
    2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
  • 2021
This work addresses the problem of shaping elasto-plastic dough by proposing to use a novel elastic end- effector to roll dough in a reinforcement learning framework, and shows that estimating stiffness using the soft end-effector can be used to effectively initialize models.

Continuous Perception for Classifying Shapes and Weights of Garmentsfor Robotic Vision Applications

The findings suggest that a modified AlexNet-LSTM architecture has the best classification performance for the garment's shape and weights.



Manipulating Highly Deformable Materials Using a Visual Feedback Dictionary

This work presents a novel visual feedback dictionary-based method for manipulating deformable objects towards a desired configuration based on visual servoing and uses an efficient technique to extract key features from the RGB sensor stream in the form of a histogram of deformable model features.

VisuoSpatial Foresight for Multi-Step, Multi-Task Fabric Manipulation

The Visual Foresight framework is extended to learn fabric dynamics that can be efficiently reused to accomplish different fabric manipulation tasks with a single goal-conditioned policy, and it is found that leveraging depth significantly improves performance.

Deep Imitation Learning of Sequential Fabric Smoothing From an Algorithmic Supervisor

In 180 physical experiments with the da Vinci Research Kit (dVRK) surgical robot, RGBD policies trained in simulation attain coverage of 83% to 95% depending on difficulty tier, suggesting that effective fabric smoothing policies can be learned from an algorithmic supervisor and that depth sensing is a valuable addition to color alone.

Learning Predictive Representations for Deformable Objects Using Contrastive Estimation

This work proposes a new learning framework that jointly optimizes both the visual representation model and the dynamics model using contrastive estimation and transfers its visual manipulation policies trained on data purely collected in simulation to a real PR2 robot through domain randomization.

Cloth Manipulation Using Random-Forest-Based Imitation Learning

This work uses a random-forest-based controller that maps the observed visual features of the cloth to an optimal control action of the manipulator and exhibits superior robustness to observation noise compared with other techniques such as convolutional neural networks and nearest neighbor searches.

Deep Transfer Learning of Pick Points on Fabric for Robot Bed-Making

This work considers the task of bed-making, where a robot sequentially grasps and pulls at pick points to increase blanket coverage, and suggests that transfer-invariant robot pick points on fabric can be effectively learned.

Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation

Dense Object Nets are presented, which build on recent developments in self-supervised dense descriptor learning, as a consistent object representation for visual understanding and manipulation and are demonstrated they can be trained quickly for a wide variety of previously unseen and potentially non-rigid objects.

Learning to Manipulate Deformable Objects without Demonstrations

This paper proposes an iterative pick-place action space that encodes the conditional relationship between picking and placing on deformable objects and obtains an order of magnitude faster learning compared to independent action-spaces on a suite of deformable object manipulation tasks with visual RGB observations.

Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassembly

This work proposes to formulate the kit assembly task as a shape matching problem, where the goal is to learn a shape descriptor that establishes geometric correspondences between object surfaces and their target placement locations from visual input.

Self-Supervised Visual Descriptor Learning for Dense Correspondence

A new approach to learning visual descriptors for dense correspondence estimation is advocated in which the power of a strong three-dimensional generative model is harnessed to automatically label correspondences in RGB-D video data.