Synthesizing 3D Shapes via Modeling Multi-view Depth Maps and Silhouettes with Deep Generative Networks

@article{Soltani2017Synthesizing3S,
  title={Synthesizing 3D Shapes via Modeling Multi-view Depth Maps and Silhouettes with Deep Generative Networks},
  author={Amir Arsalan Soltani and Haibin Huang and Jiajun Wu and Tejas D. Kulkarni and Joshua B. Tenenbaum},
  journal={2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2017},
  pages={2511-2519}
}
We study the problem of learning generative models of 3D shapes. Voxels or 3D parts have been widely used as the underlying representations to build complex 3D shapes, however, voxel-based representations suffer from high memory requirements, and parts-based models require a large collection of cached or richly parametrized parts. We take an alternative approach: learning a generative model over multi-view depth maps or their corresponding silhouettes, and using a deterministic rendering… 

Figures and Tables from this paper

Improved Modeling of 3D Shapes with Multi-view Depth Maps
TLDR
The quantitative evaluations show that the simple encoder-decoder framework is superior to existing depth map methods for reconstructing and synthesizing 3D objects and is competitive with other representations, such as point clouds, voxel grids, and implicit functions.
Learning Single-Image 3D Reconstruction by Generative Modelling of Shape, Pose and Shading
TLDR
A unified framework tackling two problems: class-specific 3D reconstruction from a single image, and generation of new 3D shape samples, that can learn to generate and reconstruct concave object classes and supports concave classes such as bathtubs and sofas, which methods based on silhouettes cannot learn.
Learning Generative Models of Shape Handles
We present a generative model to synthesize 3D shapes as sets of handles -- lightweight proxies that approximate the original 3D shape -- for applications in interactive editing, shape parsing, and
Matryoshka Networks: Predicting 3D Geometry via Nested Shape Layers
TLDR
Novel, efficient 2D encodings for 3D geometry enable reconstructing full 3D shapes from a single image at high resolution, and clearly outperform previous octree-based approaches despite having a much simpler architecture using standard network components.
Multi-chart generative surface modeling
TLDR
A 3D shape generative model based on deep neural networks that learns the shape distribution and is able to generate novel shapes, interpolate shapes, and explore the generated shape space for human body and bone (teeth) shape generation is introduced.
Leveraging 2D Data to Learn Textured 3D Mesh Generation
TLDR
This work presents the first generative model of textured 3D meshes, and introduces a new generation process that guarantees no self-intersections arise, based on the physical intuition that faces should push one another out of the way as they move.
Inverse Graphics: Unsupervised Learning of 3D Shapes from Single Images
TLDR
This paper uses a generative model that produces 2-D images as projections of a latent 3D voxel grid to use as a variational auto-encoder or using adversarial methods to learn 3D shape and pose from general datasets in good quality.
3D Object Dense Reconstruction from a Single Depth View
TLDR
The key idea is to combine the generative capabilities of autoencoders and the conditional Generative Adversarial Networks framework, to infer accurate and fine-grained 3D structures of objects in high-dimensional voxel space.
MarrNet: 3D Shape Reconstruction via 2.5D Sketches
TLDR
This work proposes MarrNet, an end-to-end trainable model that sequentially estimates 2.5D sketches and 3D object shape and derives differentiable projective functions from 3D shape to 2.
MarrNet : 3 D Shape Reconstruction via 2 . 5 D Sketches
TLDR
This work proposes MarrNet, an end-to-end trainable model that sequentially estimates 2.5D sketches and 3D object shape and derives differentiable projective functions from 3D shape to 2.
...
...

References

SHOWING 1-10 OF 35 REFERENCES
3D ShapeNets: A deep representation for volumetric shapes
TLDR
This work proposes to represent a geometric 3D shape as a probability distribution of binary variables on a 3D voxel grid, using a Convolutional Deep Belief Network, and shows that this 3D deep representation enables significant performance improvement over the-state-of-the-arts in a variety of tasks.
Analysis and synthesis of 3D shape families via deep‐learned generative models of surfaces
TLDR
Qualitative and quantitative evaluations of the method for joint analysis and synthesis of geometrically diverse 3D shape families demonstrate superior correspondence and segmentation results than previous state‐of‐the‐art approaches.
3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction
TLDR
The 3D-R2N2 reconstruction framework outperforms the state-of-the-art methods for single view reconstruction, and enables the 3D reconstruction of objects in situations when traditional SFM/SLAM methods fail (because of lack of texture and/or wide baseline).
Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision
TLDR
An encoder-decoder network with a novel projection loss defined by the projective transformation enables the unsupervised learning using 2D observation without explicit 3D supervision and shows superior performance and better generalization ability for 3D object reconstruction when the projection loss is involved.
Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
TLDR
A novel framework, namely 3D Generative Adversarial Network (3D-GAN), which generates 3D objects from a probabilistic space by leveraging recent advances in volumetric convolutional networks and generative adversarial nets, and a powerful 3D shape descriptor which has wide applications in 3D object recognition.
Multi-view Convolutional Neural Networks for 3D Shape Recognition
TLDR
This work presents a standard CNN architecture trained to recognize the shapes' rendered views independently of each other, and shows that a 3D shape can be recognized even from a single view at an accuracy far higher than using state-of-the-art3D shape descriptors.
ShapeSynth: Parameterizing model collections for coupled shape exploration and synthesis
TLDR
Unorganized collections of 3D models are analyzed to facilitate explorative shape synthesis by providing high‐level feedback of possible synthesizable shapes by jointly analyzing arrangements and shapes of parts across models.
Multi-view 3D Models from Single Images with a Convolutional Network
TLDR
A convolutional network capable of inferring a 3D representation of a previously unseen object given a single image of this object and several depth maps fused together give a full point cloud of the object.
Transformation-Grounded Image Generation Network for Novel 3D View Synthesis
We present a transformation-grounded image generation network for novel 3D view synthesis from a single image. Our approach first explicitly infers the parts of the geometry visible both in the input
Volumetric and Multi-view CNNs for Object Classification on 3D Data
TLDR
This paper introduces two distinct network architectures of volumetric CNNs and examines multi-view CNNs, providing a better understanding of the space of methods available for object classification on 3D data.
...
...