3D Shape Perception from Monocular Vision, Touch, and Shape Priors

@article{Wang20183DSP,
  title={3D Shape Perception from Monocular Vision, Touch, and Shape Priors},
  author={Shaoxiong Wang and Jiajun Wu and Xingyuan Sun and Wenzhen Yuan and William T. Freeman and Joshua B. Tenenbaum and Edward H. Adelson},
  journal={2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  year={2018},
  pages={1606-1613}
}
  • Shaoxiong Wang, Jiajun Wu, +4 authors E. Adelson
  • Published 9 August 2018
  • Computer Science
  • 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
Perceiving accurate 3D object shape is important for robots to interact with the physical world. Current research along this direction has been primarily relying on visual observations. Vision, however useful, has inherent limitations due to occlusions and the 2D-3D ambiguities, especially for perception with a monocular camera. In contrast, touch gets precise local shape information, though its efficiency for reconstructing the entire shape could be low. In this paper, we propose a novel… Expand
Efficient shape mapping through dense touch and vision
TLDR
This work proposes an incremental shape mapping method using a GelSight tactile sensor and a depth camera, and builds an implicit surface representation of the object through efficient inference on a spatial factor graph informed by a Gaussian process. Expand
Tactile Object Pose Estimation from the First Touch with Geometric Contact Rendering
TLDR
An approach to tactile pose estimation from the first touch for known objects, which provides high accuracy pose estimations from distinctive tactile observations while regressing pose distributions to account for those contact shapes that could result from different object poses. Expand
CLASP: Constrained Latent Shape Projection for Refining Object Shape from Robot Contact
TLDR
This approach consists of a shape completion network that generates a prior from RGBD data and a procedure to generate shapes consistent with both the network prior and robot contact observations, which consistently decreases the Chamfer Distance between the predicted and ground truth scenes, while other approaches do not benefit from contact information. Expand
Multi-Modal Geometric Learning for Grasping and Manipulation
TLDR
This work provides an architecture that incorporates depth and tactile information to create rich and accurate 3D models useful for robotic manipulation tasks through the use of a 3D convolutional neural network (CNN). Expand
Exploiting Visual-Outer Shape for Tactile-Inner Shape Estimation of Objects Covered with Soft Materials
TLDR
An active tactile exploration framework that can utilize the visual-outer shape to efficiently estimate the inner shape of objects covered with soft materials is proposed that jointly models the implicit surfaces of inner-outer shapes with their similarity by Gaussian processes. Expand
Learning Shape Priors for Single-View 3D Completion and Reconstruction
TLDR
The proposed ShapeHD pushes the limit of single-view shape completion and reconstruction by integrating deep generative models with adversarially learned shape priors, penalizing the model only if its output is unrealistic, not if it deviates from the ground truth. Expand
Seeing Through your Skin: Recognizing Objects with a Novel Visuotactile Sensor
TLDR
The ability of the See-Through-your-Skin sensor to classify household objects, recognize fine textures, and infer their physical properties both through numerical simulations and experiments with a smart countertop prototype are validated. Expand
Simultaneous Tactile Exploration and Grasp Refinement for Unknown Objects
TLDR
This letter proposes a grasp exploration approach using a probabilistic representation of shape, based on Gaussian Process Implicit Surfaces, which enables initial partial vision data to be augmented with additional data from successive tactile glances to refine grasp configurations. Expand
Mental geometry of 3D size and shape perception
TLDR
It is discussed how assuming that images are preserving the continuity, collinearity, and convergence invariances of projective geometry, supplements the Generic Viewpoint assumption, and simply explains other illusions, such as Ames’ Chair. Expand
Grasp State Assessment of Deformable Objects Using Visual-Tactile Fusion Perception
TLDR
The experimental results demonstrate that the C3D-VTFN is accurate and efficient enough for grasp state assessment, which can be widely applied to automatic force control, adaptive grasping, and other visual-tactile spatiotemporal sequence learning problems. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 50 REFERENCES
Enhancing visual perception of shape through tactile glances
TLDR
A probabilistic approach for learning object models based on visual and tactile perception through physical interaction with an object that is capable of choosing a small number of touches to construct object models similar to real object shapes and to determine similarities among acquired models. Expand
Localizing the object contact through matching tactile features with visual map
TLDR
A feature-based measurement model and Gaussian based motion model are built and a tactile array sensor is utilized to generate tactile images during interaction with objects and the results have proven the feasibility of the proposed framework. Expand
Structured Prediction of Unobserved Voxels from a Single Depth Image
TLDR
This work proposes an algorithm that can complete the unobserved geometry of tabletop-sized objects, based on a supervised model trained on already available volumetric elements, that maps from a local observation in a single depth image to an estimate of the surface shape in the surrounding neighborhood. Expand
MarrNet: 3D Shape Reconstruction via 2.5D Sketches
TLDR
This work proposes MarrNet, an end-to-end trainable model that sequentially estimates 2.5D sketches and 3D object shape and derives differentiable projective functions from 3D shape to 2. Expand
Three-dimensional object reconstruction of symmetric objects by fusing visual and tactile sensing
TLDR
The main contribution of this work is an optimal estimation approach for the fusion of visual and tactile data applying the constraint of object symmetry. Expand
Iterative Closest Labeled Point for tactile object shape recognition
TLDR
An extensive evaluation study with 20 real objects shows that the proposed iCLAP approach outperforms those using either of the separate sensing modalities, with a substantial recognition rate improvement of up to 18%. Expand
Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling
TLDR
A novel model is designed that simultaneously performs 3D reconstruction and pose estimation; this multi-task learning approach achieves state-of-the-art performance on both tasks. Expand
3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction
TLDR
The 3D-R2N2 reconstruction framework outperforms the state-of-the-art methods for single view reconstruction, and enables the 3D reconstruction of objects in situations when traditional SFM/SLAM methods fail (because of lack of texture and/or wide baseline). Expand
Learning Shape Priors for Single-View 3D Completion and Reconstruction
TLDR
The proposed ShapeHD pushes the limit of single-view shape completion and reconstruction by integrating deep generative models with adversarially learned shape priors, penalizing the model only if its output is unrealistic, not if it deviates from the ground truth. Expand
Shape completion enabled robotic grasping
TLDR
This work provides an architecture to enable robotic grasp planning via shape completion through the use of a 3D convolutional neural network trained on a new open source dataset of over 440,000 3D exemplars captured from varying viewpoints. Expand
...
1
2
3
4
5
...