• Corpus ID: 226226846

Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs

@article{Pan2021Do2G,
  title={Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs},
  author={Xingang Pan and Bo Dai and Ziwei Liu and Chen Change Loy and Ping Luo},
  journal={ArXiv},
  year={2021},
  volume={abs/2011.00844}
}
Natural images are projections of 3D objects on a 2D image plane. While state-of-the-art 2D generative models like GANs show unprecedented quality in modeling the natural image manifold, it is unclear whether they implicitly capture the underlying 3D object structures. And if so, how could we exploit such knowledge to recover the 3D shapes of objects in the images? To answer these questions, in this work, we present the first attempt to directly mine 3D geometric clues from an off-the-shelf 2D… 
Lifting 2D StyleGAN for 3D-Aware Face Generation
TLDR
Qualitative and quantitative results show the superiority of the approach over existing methods on 3D-controllable GANs in content controllability while generating realistic high quality images.
GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds
TLDR
GANcraft is presented, an unsupervised neural rendering framework for generating photorealistic images of large 3D block worlds such as those created in Minecraft, and allows user control over both scene semantics and output style.
Ensembling with Deep Generative Views
TLDR
This work uses StyleGAN2 as the source of generative augmentations and investigates whether such views can be applied to real images to benefit downstream analysis tasks such as image classification.
Towards Open-World Text-Guided Face Image Generation and Manipulation
TLDR
This work proposes a unified framework for both face image generation and manipulation that produces diverse and high-quality images with an unprecedented resolution at 1024 from multimodal inputs and supports open-world scenarios, including both image and text, without any re-training, fine-tuning, or post-processing.
Labels4Free: Unsupervised Segmentation using StyleGAN
TLDR
This work proposes to augment the StyleGAN2 generator architecture with a segmentation branch and to split the generator into a foreground and background network, allowing it to generate soft segmentation masks for the foreground object in an unsupervised fashion.
Disentangling Autoencoders (DAE)
TLDR
A novel, non-probabilistic disentangling framework for autoencoders, based on the principles of symmetry transformations in group-theory, that can have better disentanglement when variances of each features are different is proposed.
FreeStyleGAN: Free-view Editable Portrait Rendering with the Camera Manifold
Fig. 1. We introduce a new approach that generates an image with StyleGAN defined by a precise 3D camera. This enables faces synthesized with StyleGAN to be used in 3D free-viewpoint rendering, while
Self-Supervised 3D Mesh Reconstruction from Single Images
TLDR
This paper proposes a Self-supervised Mesh Reconstruction (SMR) approach to enhance 3D mesh attribute learning process, motivated by observations that 3D attributes from interpolation and prediction should be consistent, and feature representation of landmarks from all imagesShould be consistent.
AUV-Net: Learning Aligned UV Maps for Texture Transfer and Synthesis
TLDR
AUV-Net is proposed which learns to embed 3D surfaces into a 2D aligned UV space, by mapping the corresponding semantic parts of different 3D shapes to the same location in the UV space and can thus be easily synthesized by generative models of images.
Feature Sharing Attention 3D Face Reconstruction with Unsupervised Learning from In-the-Wild Photo Collection
  • Xiaoxiao Yang
  • Computer Science
    Journal of Physics: Conference Series
  • 2022
TLDR
A3D face reconstruction algorithm that takes a single face image as input and uses a encoder-decoder network to predict the 3D face and has a certain improvement in the two evaluation indexes of Scale Invariant Depth Error and Mean Angle Deviation.
...
1
2
3
...

References

SHOWING 1-10 OF 54 REFERENCES
Self-supervised Single-view 3D Reconstruction via Semantic Consistency
TLDR
This work is the first to try and solve the single-view reconstruction problem without a category-specific template mesh or semantic keypoints, and demonstrates that the unsupervised method performs comparably if not better than existing category- specific reconstruction methods learned with supervision.
HoloGAN: Unsupervised Learning of 3D Representations From Natural Images
TLDR
HoloGAN is the first generative model that learns 3D representations from natural images in an entirely unsupervised manner and is shown to be able to generate images with similar or higher visual quality than other generative models.
Shape and Viewpoint without Keypoints
We present a learning framework that learns to recover the 3D shape, pose and texture from a single image, trained on an image collection without any ground truth 3D shape, multi-view, camera
Unsupervised Generative 3D Shape Learning from Natural Images
TLDR
This paper presents the first method to learn a generative model of 3D shapes from natural images in a fully unsupervised way, and demonstrates that this method can learn realistic 3D shape of faces by using only the natural images of the FFHQ dataset.
Learning Category-Specific Mesh Reconstruction from Image Collections
TLDR
A learning framework for recovering the 3D shape, camera, and texture of an object from a single image by incorporating texture inference as prediction of an image in a canonical appearance space and shows that semantic keypoints can be easily associated with the predicted shapes.
Implicit Mesh Reconstruction from Unannotated Image Collections
TLDR
An approach to infer the 3D shape, texture, and camera pose for an object from a single RGB image, using only category-level image collections with foreground masks as supervision, and qualitatively demonstrates its applicability over a set of about 30 object categories.
Lifting AutoEncoders: Unsupervised Learning of a Fully-Disentangled 3D Morphable Model Using Deep Non-Rigid Structure From Motion
TLDR
This work introduces Lifting Autoencoders, a generative 3D surface-based model of object categories that can be controlled in terms of interpretable geometry and appearance factors, allowing it to perform photorealistic image manipulation of identity, expression, 3D pose, and illumination properties.
Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
TLDR
A novel framework, namely 3D Generative Adversarial Network (3D-GAN), which generates 3D objects from a probabilistic space by leveraging recent advances in volumetric convolutional networks and generative adversarial nets, and a powerful 3D shape descriptor which has wide applications in 3D object recognition.
Learning to Regress 3D Face Shape and Expression From an Image Without 3D Supervision
TLDR
To train a network without any 2D-to-3D supervision, RingNet is presented, which learns to compute 3D face shape from a single image and achieves invariance to expression by representing the face using the FLAME model.
End-to-End Recovery of Human Shape and Pose
TLDR
This work introduces an adversary trained to tell whether human body shape and pose parameters are real or not using a large database of 3D human meshes, and produces a richer and more useful mesh representation that is parameterized by shape and 3D joint angles.
...
1
2
3
4
5
...