Self-supervised Transparent Liquid Segmentation for Robotic Pouring

  title={Self-supervised Transparent Liquid Segmentation for Robotic Pouring},
  author={Gautham Narayan Narasimhan and Kai Zhang and Ben Eisner and Xingyu Lin and David Held},
Liquid state estimation is important for robotics tasks such as pouring; however, estimating the state of transparent liquids is a challenging problem. We propose a novel segmentation pipeline that can segment transparent liquids such as water from a static, RGB image without requiring any manual annotations or heating of the liquid for training. Instead, we use a generative model that is capable of translating images of colored liquids into synthetically generated transparent liquid images… 

Figures and Tables from this paper


Contrastive Learning for Unpaired Image-to-Image Translation
The framework enables one-sided translation in the unpaired image-to-image translation setting, while improving quality and reducing training time, and can be extended to the training setting where each "domain" is only a single image.
Segmenting Transparent Objects in the Wild
A novel boundary-aware segmentation method, termed TransLab, is proposed, which exploits boundary as the clue to improve segmentation of transparent objects and significantly outperforms 20 recent object segmentation methods based on deep learning.
RGB-D Local Implicit Function for Depth Completion of Transparent Objects
A new approach for depth completion of transparent objects from a single RGB-D image using a local implicit neural representation built on ray-voxel pairs that allows the method to generalize to unseen objects and achieve fast inference speed.
SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation
SoftGym is presented, a set of open-source simulated benchmarks for manipulating deformable objects, with a standard OpenAI Gym API and a Python interface for creating new environments, to enable reproducible research in this important area.
Transparent object segmentation from casually captured videos
A from‐coarse‐to‐fine transparent object segmentation method, which utilizes trajectory clustering to roughly distinguish the transparent from the background and refine the segmentation based on combination information of color and distortion.
Deep Polarization Cues for Transparent Object Segmentation
This paper reframes the problem of transparent object segmentation into the realm of light polarization, i.e., the rotation of light waves, using a polarization camera to capture multi-modal imagery and couple this with a unique deep learning backbone for processing polarization input data.
Inferring the Material Properties of Granular Media for Robotic Tasks
This work presents a software and hardware framework that automatically calibrates a fast physics simulator to accurately simulate granular materials by inferring material properties from real-world depth images of granular formations (i.e., piles and rings).
Multi-Modal Transfer Learning for Grasping Transparent and Specular Objects
This work introduces a method for learning a multi-modal perception model by bootstrapping from an existing uni- modal model, foregoing the need for ground-truth grasp success labels nor real grasp attempts.
Precision Pouring into Unknown Containers by Service Robots
Two approaches for controlling the motion of a service robot as it pours liquid precisely from an unknown container into another unknown container without the need of any external tools are proposed.
KeyPose: Multi-View 3D Labeling and Keypoint Estimation for Transparent Objects
This paper establishes an easy method for capturing and labeling 3D keypoints on desktop objects with an RGB camera, and develops a deep neural network, called KeyPose, that learns to accurately predict object poses using 3DKeypoints, from stereo input, and works even for transparent objects.