• Corpus ID: 219401636

Learning Neural Light Transport

  title={Learning Neural Light Transport},
  author={Paul Sanzenbacher and Lars M. Mescheder and Andreas Geiger},
In recent years, deep generative models have gained significance due to their ability to synthesize natural-looking images with applications ranging from virtual reality to data augmentation for training computer vision models. While existing models are able to faithfully learn the image distribution of the training set, they often lack controllability as they operate in 2D pixel space and do not model the physical image formation process. In this work, we investigate the importance of 3D… 
Neural scene graph rendering
We present a neural scene graph---a modular and controllable representation of scenes with elements that are learned from data. We focus on the forward rendering problem, where the scene graph is


Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis
This work defines the new task of 3D controllable image synthesis and proposes an approach for solving it by reasoning both in 3D space and in the 2D image domain, and demonstrates that the model is able to disentangle latent 3D factors of simple multi-object scenes in an unsupervised fashion from raw images.
Deep‐learning the Latent Space of Light Transport
This work suggests a two‐stage operator comprising a 3D network that first transforms the point cloud into a latent representation, which is later on projected to the 2D output image using a dedicated 3D‐2D network in a second step.
Geometric Image Synthesis
This work proposes a trainable, geometry-aware image generation method that leverages various types of scene information, including geometry and segmentation, to create realistic looking natural images that match the desired scene structure.
Deferred Neural Rendering: Image Synthesis using Neural Textures
This work proposes Neural Textures, which are learned feature maps that are trained as part of the scene capture process that can be utilized to coherently re-render or manipulate existing video content in both static and dynamic environments at real-time rates.
HoloGAN: Unsupervised Learning of 3D Representations From Natural Images
HoloGAN is the first generative model that learns 3D representations from natural images in an entirely unsupervised manner and is shown to be able to generate images with similar or higher visual quality than other generative models.
Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer
A differentiable rendering framework which allows gradients to be analytically computed for all pixels in an image and to view foreground rasterization as a weighted interpolation of local properties and background rasterized as a distance-based aggregation of global geometry.
Texture Fields: Learning Texture Representations in Function Space
Texture Fields, a novel texture representation which is based on regressing a continuous 3D function parameterized with a neural network is proposed, which is able to represent high frequency texture and naturally blend with modern deep learning techniques.
Occupancy Networks: Learning 3D Reconstruction in Function Space
This paper proposes Occupancy Networks, a new representation for learning-based 3D reconstruction methods that encodes a description of the 3D output at infinite resolution without excessive memory footprint, and validate that the representation can efficiently encode 3D structure and can be inferred from various kinds of input.
View Synthesis by Appearance Flow
This work addresses the problem of novel view synthesis: given an input image, synthesizing new images of the same object or scene observed from arbitrary viewpoints and shows that for both objects and scenes, this approach is able to synthesize novel views of higher perceptual quality than previous CNN-based techniques.
Augmented Reality Meets Computer Vision: Efficient Data Generation for Urban Driving Scenes
This work proposes an alternative paradigm which combines real and synthetic data for learning semantic instance segmentation and object detection models, and introduces a novel dataset of augmented urban driving scenes with 360 degree images that are used as environment maps to create realistic lighting and reflections on rendered objects.