Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis

@article{Liao2020TowardsUL,
  title={Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis},
  author={Yiyi Liao and K. Schwarz and Lars M. Mescheder and Andreas Geiger},
  journal={2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2020},
  pages={5870-5879}
}
In recent years, Generative Adversarial Networks have achieved impressive results in photorealistic image synthesis. This progress nurtures hopes that one day the classical rendering pipeline can be replaced by efficient models that are learned directly from images. However, current image synthesis models operate in the 2D domain where disentangling 3D properties such as camera viewpoint or object pose is challenging. Furthermore, they lack an interpretable and controllable representation. Our… Expand
CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields
TLDR
This work learns a 3Dand camera-aware generative model which faithfully recovers not only the image but also the camera data distribution, and proposes to decompose the scene into a background and foreground model, leading to more efficient and disentangled scene representations. Expand
Learning Neural Light Transport
TLDR
This work presents an approach for learning light transport in static and dynamic 3D scenes using a neural network with the goal of predicting photorealistic images, and reasons in both 3D and 2D space, thus enabling global illumination effects and manipulation of 3D scene geometry. Expand
Unsupervised Learning of Depth and Depth-of-Field Effect from Natural Images with Aperture Rendering Generative Adversarial Networks
TLDR
DoF mixture learning is developed, which enables the generator to learn real image distribution while generating diverse DoF images and devise a center focus prior to guiding the learning direction to address the ambiguities triggered by unsupervised setting. Expand
GaussiGAN: Controllable Image Synthesis with 3D Gaussians from Unposed Silhouettes
TLDR
This work presents an algorithm that learns a coarse 3D representation of objects from unposed multi-view 2D mask supervision, then uses it to generate detailed mask and image texture, and helps move towards structured representations that handle more real-world variation in learningbased object reconstruction. Expand
Unsupervised Novel View Synthesis from a Single Image
TLDR
This work pre-train a purely generative decoder model using a GAN formulation while at the same time training an encoder network to invert the mapping from latent code to images and shows that the framework achieves results comparable to the state of the art on ShapeNet. Expand
Realistic Image Synthesis with Configurable 3D Scene Layouts
Recent conditional image synthesis approaches provide high-quality synthesized images. However, it is still challenging to accurately adjust image contents such as the positions and orientations ofExpand
Learning Implicit Surface Light Fields
TLDR
This work proposes a novel implicit representation for capturing the visual appearance of an object in terms of its surface light field and shows that the proposed representation can be embedded into a variational auto-encoder for generating novel appearances that conform to the specified illumination conditions. Expand
Inverse Graphics GAN: Learning to Generate 3D Shapes from Unstructured 2D Data
TLDR
This paper introduces the first scalable training technique for 3D generative models from 2D data which utilizes an off-the-shelf non-differentiable renderer and proposes discriminator output matching to ensure that the neural renderer learns to smooth over the rasterization appropriately. Expand
Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision
TLDR
This work proposes a differentiable rendering formulation for implicit shape and texture representations, showing that depth gradients can be derived analytically using the concept of implicit differentiation, and finds that this method can be used for multi-view 3D reconstruction, directly resulting in watertight meshes. Expand
InSeGAN: A Generative Approach to Segmenting Identical Instances in Depth Images
In this paper, we present InSeGAN, an unsupervised 3D generative adversarial network (GAN) for segmenting (nearly) identical instances of rigid objects in depth images. Using an analysis-by-synthesisExpand
...
1
2
3
...

References

SHOWING 1-10 OF 54 REFERENCES
Visual Object Networks: Image Generation with Disentangled 3D Representations
TLDR
A new generative model, Visual Object Networks (VONs), synthesizing natural images of objects with a disentangled 3D representation that enables many 3D operations such as changing the viewpoint of a generated image, shape and texture editing, linear interpolation in texture and shape space, and transferring appearance across different objects and viewpoints. Expand
HoloGAN: Unsupervised Learning of 3D Representations From Natural Images
TLDR
HoloGAN is the first generative model that learns 3D representations from natural images in an entirely unsupervised manner and is shown to be able to generate images with similar or higher visual quality than other generative models. Expand
DeepVoxels: Learning Persistent 3D Feature Embeddings
TLDR
This work proposes DeepVoxels, a learned representation that encodes the view-dependent appearance of a 3D scene without having to explicitly model its geometry, based on a Cartesian 3D grid of persistent embedded features that learn to make use of the underlying3D scene structure. Expand
Geometric Image Synthesis
TLDR
This work proposes a trainable, geometry-aware image generation method that leverages various types of scene information, including geometry and segmentation, to create realistic looking natural images that match the desired scene structure. Expand
Photographic Image Synthesis with Cascaded Refinement Networks
  • Qifeng Chen, V. Koltun
  • Computer Science
  • 2017 IEEE International Conference on Computer Vision (ICCV)
  • 2017
TLDR
It is shown that photographic images can be synthesized from semantic layouts by a single feedforward network with appropriate structure, trained end-to-end with a direct regression objective. Expand
Learning What and Where to Draw
TLDR
This work proposes a new model, the Generative Adversarial What-Where Network (GAWWN), that synthesizes images given instructions describing what content to draw in which location, and shows high-quality 128 x 128 image synthesis on the Caltech-UCSD Birds dataset. Expand
Generative Image Modeling Using Style and Structure Adversarial Networks
TLDR
This paper factorize the image generation process and proposes Style and Structure Generative Adversarial Network, a model that is interpretable, generates more realistic images and can be used to learn unsupervised RGBD representations. Expand
A Layer-Based Sequential Framework for Scene Generation with GANs
TLDR
This work presents a scene generation framework based on Generative Adversarial Networks (GANs) to sequentially compose a scene, breaking down the underlying problem into smaller ones. Expand
Structured3D: A Large Photo-realistic Dataset for Structured 3D Modeling
TLDR
This paper presents a new synthetic dataset, Structured3D, with the aim of providing large-scale photo-realistic images with rich 3D structure annotations for a wide spectrum of structured 3D modeling tasks, and takes advantage of the availability of professional interior designs to automatically extract 3D structures from them. Expand
Image Generation From Layout
TLDR
The proposed Layout2Im model significantly outperforms the previous state of the art, boosting the best reported inception score by 24.66% and 28.57% on the very challenging COCO-Stuff and Visual Genome datasets, respectively. Expand
...
1
2
3
4
5
...