Corpus ID: 220364071

GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis

@article{Schwarz2020GRAFGR,
  title={GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis},
  author={Katja Schwarz and Yiyi Liao and Michael Niemeyer and Andreas Geiger},
  journal={ArXiv},
  year={2020},
  volume={abs/2007.02442}
}
While 2D generative adversarial networks have enabled high-resolution image synthesis, they largely lack an understanding of the 3D world and the image formation process. Thus, they do not provide precise control over camera viewpoint or object pose. To address this problem, several recent approaches leverage intermediate voxel-based representations in combination with differentiable rendering. However, existing methods either produce low image resolution or fall short in disentangling camera… Expand
Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications
TLDR
An overview of GANs with a special focus on algorithms and applications for visual synthesis is provided and several important techniques to stabilize GAN training are covered, which has a reputation for being notoriously difficult. Expand
GRF: Learning a General Radiance Field for 3D Scene Representation and Rendering
TLDR
The key to the approach is to explicitly integrate the principle of multi- view geometry to obtain the internal representations from observed 2D views, guaranteeing the learned implicit representations meaningful and multi-view consistent. Expand
Unconstrained Scene Generation with Locally Conditioned Radiance Fields
TLDR
Generative Scene Networks is introduced, which learns to decompose scenes into a collection of many local radiance fields that can be rendered from a free moving camera, and which produces quantitatively higher-quality scene renderings across several different scene datasets. Expand
pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis
TLDR
This work proposes a novel generative model, named Periodic Implicit Generative Adversarial Networks ($\pi$-GAN or pi-GAN), for high-quality 3D-aware image synthesis that leverages neural representations with periodic activation functions and volumetric rendering to represent scenes as view-consistent 3D representations with fine detail. Expand
Editing Conditional Radiance Fields
TLDR
This paper introduces a method for propagating coarse 2D user scribbles to the 3D space, to modify the color or shape of a local region, and proposes a conditional radiance field that incorporates new modular network components, including a shape branch that is shared across object instances. Expand
Mixture of volumetric primitives for efficient neural rendering
TLDR
Mixture of Volumetric Primitives (MVP), a representation for rendering dynamic 3D content that combines the completeness of volumetric representations with the efficiency of primitive-based rendering, is presented. Expand
RGB-D Local Implicit Function for Depth Completion of Transparent Objects
TLDR
A new approach for depth completion of transparent objects from a single RGB-D image using a local implicit neural representation built on ray-voxel pairs that allows the method to generalize to unseen objects and achieve fast inference speed. Expand
StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis
  • Jiatao Gu, Lingjie Liu, Peng Wang, C. Theobalt
  • Computer Science, Mathematics
  • ArXiv
  • 2021
We propose StyleNeRF, a 3D-aware generative model for photo-realistic highresolution image synthesis with high multi-view consistency, which can be trained on unstructured 2D images. ExistingExpand
3D Scene Compression through Entropy Penalized Neural Representation Functions
TLDR
This work significantly outperforms a state-of-the-art conventional approach for scene compression, achieving simultaneously higher quality reconstructions and lower bitrates, and shows that the performance at lower bitrate can be improved by jointly representing multiple scenes using a soft form of parameter sharing. Expand
A-NeRF: Surface-free Human 3D Pose Refinement via Neural Rendering
Fig. 1. Our A-NeRF test-time optimization for monocular 3D human pose estimation jointly learns a volumetric body model of the user that can be animated and works with diverse body shapes (left),Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 85 REFERENCES
Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis
TLDR
This work defines the new task of 3D controllable image synthesis and proposes an approach for solving it by reasoning both in 3D space and in the 2D image domain, and demonstrates that the model is able to disentangle latent 3D factors of simple multi-object scenes in an unsupervised fashion from raw images. Expand
Visual Object Networks: Image Generation with Disentangled 3D Representations
TLDR
A new generative model, Visual Object Networks (VONs), synthesizing natural images of objects with a disentangled 3D representation that enables many 3D operations such as changing the viewpoint of a generated image, shape and texture editing, linear interpolation in texture and shape space, and transferring appearance across different objects and viewpoints. Expand
HoloGAN: Unsupervised Learning of 3D Representations From Natural Images
TLDR
HoloGAN is the first generative model that learns 3D representations from natural images in an entirely unsupervised manner and is shown to be able to generate images with similar or higher visual quality than other generative models. Expand
Monocular Neural Image Based Rendering With Continuous View Control
  • Jie Song, Xu Chen, Otmar Hilliges
  • Computer Science
  • 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
  • 2019
TLDR
The experiments show that both proposed components, the transforming encoder-decoder and depth-guided appearance mapping, lead to significantly improved generalization beyond the training views and in consequence to more accurate view synthesis under continuous 6-DoF camera control. Expand
Stereo Magnification: Learning View Synthesis using Multiplane Images
TLDR
This paper explores an intriguing scenario for view synthesis: extrapolating views from imagery captured by narrow-baseline stereo cameras, including VR cameras and now-widespread dual-lens camera phones, and proposes a learning framework that leverages a new layered representation that is called multiplane images (MPIs). Expand
Geometric Image Synthesis
TLDR
This work proposes a trainable, geometry-aware image generation method that leverages various types of scene information, including geometry and segmentation, to create realistic looking natural images that match the desired scene structure. Expand
DeepVoxels: Learning Persistent 3D Feature Embeddings
TLDR
This work proposes DeepVoxels, a learned representation that encodes the view-dependent appearance of a 3D scene without having to explicitly model its geometry, based on a Cartesian 3D grid of persistent embedded features that learn to make use of the underlying3D scene structure. Expand
Pushing the Boundaries of View Extrapolation With Multiplane Images
TLDR
This paper presents a theoretical analysis showing how the range of views that can be rendered from an MPI increases linearly with the MPI disparity sampling frequency, as well as a novel MPI prediction procedure that theoretically enables view extrapolations of up to 4 times the lateral viewpoint movement allowed by prior work. Expand
Learning Implicit Surface Light Fields
TLDR
This work proposes a novel implicit representation for capturing the visual appearance of an object in terms of its surface light field and shows that the proposed representation can be embedded into a variational auto-encoder for generating novel appearances that conform to the specified illumination conditions. Expand
Pix2Vox: Context-Aware 3D Reconstruction From Single and Multi-View Images
TLDR
Experimental results on the ShapeNet and Pix3D benchmarks indicate that the proposed Pix2Vox outperforms state-of-the-arts by a large margin, and the proposed method is 24 times faster than 3D-R2N2 in terms of backward inference time. Expand
...
1
2
3
4
5
...