Learning Visual Shape Control of Novel 3D Deformable Objects from Partial-View Point Clouds

  title={Learning Visual Shape Control of Novel 3D Deformable Objects from Partial-View Point Clouds},
  author={Bao Thach and Brian Cho and Alan Kuntz and Tucker Hermans},
  journal={2022 International Conference on Robotics and Automation (ICRA)},
If robots could reliably manipulate the shape of 3D deformable objects, they could find applications in fields ranging from home care to warehouse fulfillment to surgical assistance. Analytic models of elastic, 3D deformable objects require numerous parameters to describe the potentially infinite degrees of freedom present in determining the object's shape. Previous attempts at performing 3D shape control rely on hand-crafted features to represent the object shape and require training of object… 

Lattice-based shape tracking and servoing of elastic objects

The proposed general tracking-servoing approach for controlling the shape of elastic deformable objects using robotic arms is based on the As-Rigid-As-Possible (ARAP) deformation model, which requires no mechanical parameter of the object to be known and can drive the object toward desired shapes through large deformations.

Software Architecture For Deformable Linear Object Manipulation: A Shape Manipulation Case Study

A layered software architecture consisting of a decision layer, a skill layer and a functional layer is presented to be a blueprint for a unified software architecture satisfying the requirements of robotic systems to achieve deformable linear object manipulation.

VIRDO++: Real-World, Visuo-tactile Dynamics and Perception of Deformable Objects

This work presents a novel approach for, and real-world demonstration of, multimodal visuotactile state-estimation and dynamics prediction for deformable objects via a new formulation for deformation dynamics and a complementary state-ESTimation algorithm that enables practical real- world application by removing the need for contact patches.

ToolFlowNet: Robotic Manipulation with Tools via Predicting Tool Flow from Point Clouds

This paper proposes a novel framework for learning policies from point clouds for robotic manipulation with tools using a novel neural network, ToolFlowNet, which predicts dense per-point transformation on the tool that the robot controls, and then uses the transformation to derive the transformation that the machine should execute.

Task-Oriented Grasping Position Selection in Deformable Object Manipulation

This letter develops grasping position selection methods that take into account the properties of both the task and DO and propose algorithms based on metrics, which enable robots to select appropriate grasping positions in DOM.

Learning intraoperative organ manipulation with context-based reinforcement learning

It is shown that the rlman framework can support the training of different RL algorithms for solving surgical sub-task, analysing the importance of context information for generalisation capabilities.



DeformerNet: A Deep Learning Approach to 3D Deformable Object Manipulation

A novel deep learning approach to solving the 3D shape servo problem is proposed that takes point clouds of the deformable objects as the inputs and outputs feature vectors and develops a DNN that maps these feature vectors to the desired end-effector’s Cartesian position.

3-D Deformable Object Manipulation Using Deep Neural Networks

A deep neural network based controller to servo control the position and shape of deformable objects with unknown deformation properties is presented and a novel feature to describe deformableObjects’ deformation used in visual servoing is introduced.

DefGraspSim: Simulation-based grasping of 3D deformable objects

A large-scale simulation-based study of 3D deformable object grasping is conducted, which extends analytical methods through accurate modeling of object deformation, enables safe execution of experiments, and provides full observability of performance metrics.

Learning Continuous 3D Reconstructions for Geometrically Aware Grasping

This work proposes to utilize a novel, learned 3D reconstruction to enable geometric awareness in a grasping system, and leverage the structure of the reconstruction network to learn a grasp success classifier which serves as the objective function for a continuous grasp optimization.

Fourier-Based Shape Servoing: A New Feedback Method to Actively Deform Soft Objects into Desired 2-D Image Contours

The design of a vision-based method to automatically deform soft objects into desired two-dimensional shapes with robot manipulators is addressed, with an innovative feedback representation of the object's shape based on a truncated Fourier series.

Learning to Manipulate Deformable Objects without Demonstrations

This paper proposes an iterative pick-place action space that encodes the conditional relationship between picking and placing on deformable objects and obtains an order of magnitude faster learning compared to independent action-spaces on a suite of deformable object manipulation tasks with visual RGB observations.

On the visual deformation servoing of compliant objects: Uncalibrated control methods and experiments

Two new visual servoing approaches to explicitly servo-control elastic deformations are presented, both of which provide a way to automatically control deformations in a model-free manner.

Sim-to-Real Reinforcement Learning for Deformable Object Manipulation

This work uses a combination of state-of-the-art deep reinforcement learning algorithms to solve the problem of manipulating deformable objects (specifically cloth), and evaluates the approach on three tasks --- folding a towel up to a mark, folding a face towel diagonally, and draping a piece of cloth over a hanger.

Self-supervised 6D Object Pose Estimation for Robot Manipulation

This work introduces a robot system for self-supervised 6D object pose estimation and improves object segmentation and 6D pose estimation performance, and consequently enables the system to grasp objects more reliably.