CAPNet: Continuous Approximation Projection For 3D Point Cloud Reconstruction Using 2D Supervision

@inproceedings{NavaneetK2018CAPNetCA,
  title={CAPNet: Continuous Approximation Projection For 3D Point Cloud Reconstruction Using 2D Supervision},
  author={L. NavaneetK. and Priyanka Mandikal and Mayank Agarwal and R. Venkatesh Babu},
  booktitle={AAAI Conference on Artificial Intelligence},
  year={2018}
}
Knowledge of 3D properties of objects is a necessity in order to build effective computer vision systems. However, lack of large scale 3D datasets can be a major constraint for datadriven approaches in learning such properties. We consider the task of single image 3D point cloud reconstruction, and aim to utilize multiple foreground masks as our supervisory data to alleviate the need for large scale 3D datasets. A novel differentiable projection module, called ‘CAPNet’, is introduced to obtain… 
View PDF on arXiv
47 Citations
Highly Influential Citations
3
Background Citations
22
Methods Citations
21

47 Citations

Unsupervised Learning of Fine Structure Generation for 3D Point Clouds by 2D Projection Matching

This work casts 3D point cloud learning as a 2D projection matching problem, and introduces structure adaptive sampling to randomly sample 2D points within the silhouettes as an irregular point supervision, which alleviates the consistency issue of sampling from different view angles.

From Image Collections to Point Clouds With Self-Supervised Shape and Pose Networks

A key novelty of the proposed technique is to impose 3D geometric reasoning into predicted 3D point clouds by rotating them with randomly sampled poses and then enforcing cycle consistency on both 3D reconstructions and poses.

Weakly-Supervised Single-view Dense 3D Point Cloud Reconstruction via Differentiable Renderer

This paper focuses on complete three-dimensional point cloud reconstruction based on a single red-green-blue (RGB) image, a task that cannot be approached using classical reconstruction techniques, and uses an encoder-decoder framework to encode the RGB information in latent space, and to predict the 3D structure of the considered object from different viewpoints.

Flow-based GAN for 3D Point Cloud Generation from a Single Image

This work introduces a hybrid explicit-implicit generative modeling scheme, which inherits the flow-based explicit generative models for sampling point clouds with arbitrary resolutions while improving the detailed 3D structures of point clouds by leveraging the implicit generative adversarial networks (GANs).

Visual Enhanced 3D Point Cloud Reconstruction from A Single Image

Experimental results demonstrate that the proposed method outperforms existing techniques significantly, both qualitatively and quantitatively, and has fewer training parameters.

Fast-Image2Point: Towards Real-Time Point Cloud Reconstruction of a Single Image using 3D Supervision

This study addresses current problems in reconstructing objects displayed in a single-view image in a faster (real-time) fashion by developing a simple yet powerful deep neural framework.

Towards Weakly Supervised Semantic Segmentation in 3D Graph-Structured Point Clouds of Wild Scenes

It is argued that 2D supervision is capable of providing sufficient guidance information for training 3D semantic segmentation models of natural scene point clouds while not explicitly capturing their inherent structures, even with only single view per training sample.

3D-ARNet: An accurate 3D point cloud reconstruction network from a single-image

  • Hui ChenYipeng Zuo
  • Computer Science, Environmental Science
    Multimedia Tools and Applications
  • 2021
A two-stage training network that can reconstruct point cloud from a single image is proposed, namely, 3D-ARNet, which contains a pre-trained point cloud auto-encoder which takes simple point cloud as input, and finally obtains an accurately reconstructed point cloud.

DV-Net: Dual-view network for 3D reconstruction by fusing multiple sets of gated control point clouds

Dual-View 3D Reconstruction via Learning Correspondence and Dependency of Point Cloud Regions

Qualitative and quantitative experiments demonstrate that the proposed Dual-View 3D Point Cloud reconstruction architecture named DVPC outperforms the state-of-the-art methods in terms of reconstruction quality.

References

SHOWING 1-10 OF 29 REFERENCES

3D-LMNet: Latent Embedding Matching for Accurate and Diverse 3D Point Cloud Reconstruction from a Single Image

3D-LMNet, a latent embedding matching approach for 3D reconstruction, is proposed, which outperform state-of-the-art approaches on the task of single-view3D reconstruction on both real and synthetic datasets while generating multiple plausible reconstructions, demonstrating the generalizability and utility of the approach.

A Point Set Generation Network for 3D Object Reconstruction from a Single Image

This paper addresses the problem of 3D reconstruction from a single image, generating a straight-forward form of output unorthordox, and designs architecture, loss function and learning paradigm that are novel and effective, capable of predicting multiple plausible 3D point clouds from an input image.

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

This paper designs a novel type of neural network that directly consumes point clouds, which well respects the permutation invariance of points in the input and provides a unified architecture for applications ranging from object classification, part segmentation, to scene semantic parsing.

Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction

This paper uses 2D convolutional operations to predict the 3D structure from multiple viewpoints and jointly apply geometric reasoning with 2D projection optimization, and introduces the pseudo-renderer, a differentiable module to approximate the true rendering operation, to synthesize novel depth maps for optimization.

Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision

An encoder-decoder network with a novel projection loss defined by the projective transformation enables the unsupervised learning using 2D observation without explicit 3D supervision and shows superior performance and better generalization ability for 3D object reconstruction when the projection loss is involved.

MarrNet : 3 D Shape Reconstruction via 2 . 5 D Sketches

This work proposes MarrNet, an end-to-end trainable model that sequentially estimates 2.5D sketches and 3D object shape and derives differentiable projective functions from 3D shape to 2.

3D-PSRNet: Part Segmented 3D Point Cloud Reconstruction From a Single Image

It is demonstrated that jointly training for both reconstruction and segmentation leads to improved performance in both the tasks, when compared to training for each task individually.

3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction

The 3D-R2N2 reconstruction framework outperforms the state-of-the-art methods for single view reconstruction, and enables the 3D reconstruction of objects in situations when traditional SFM/SLAM methods fail (because of lack of texture and/or wide baseline).

Multi-view Supervision for Single-View Reconstruction via Differentiable Ray Consistency

A differentiable formulation which allows computing gradients of the 3D shape given an observation from an arbitrary view is proposed by reformulating view consistency using a differentiable ray consistency (DRC) term and it is shown that this formulation can be incorporated in a learning framework to leverage different types of multi-view observations.

PointCNN

This work proposes to learn a X-transformation from the input points, which is used for simultaneously weighting the input features associated with the points and permuting them into latent potentially canonical order, and calls it PointCNN.