Corpus ID: 220364071

# GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis

@article{Schwarz2020GRAFGR,
title={GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis},
author={Katja Schwarz and Yiyi Liao and Michael Niemeyer and Andreas Geiger},
journal={ArXiv},
year={2020},
volume={abs/2007.02442}
}
While 2D generative adversarial networks have enabled high-resolution image synthesis, they largely lack an understanding of the 3D world and the image formation process. Thus, they do not provide precise control over camera viewpoint or object pose. To address this problem, several recent approaches leverage intermediate voxel-based representations in combination with differentiable rendering. However, existing methods either produce low image resolution or fall short in disentangling camera… Expand
Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications
• Computer Science
• Proceedings of the IEEE
• 2021
An overview of GANs with a special focus on algorithms and applications for visual synthesis is provided and several important techniques to stabilize GAN training are covered, which has a reputation for being notoriously difficult. Expand
GRF: Learning a General Radiance Field for 3D Scene Representation and Rendering
• Computer Science
• ArXiv
• 2020
The key to the approach is to explicitly integrate the principle of multi- view geometry to obtain the internal representations from observed 2D views, guaranteeing the learned implicit representations meaningful and multi-view consistent. Expand
Unconstrained Scene Generation with Locally Conditioned Radiance Fields
• Computer Science
• ArXiv
• 2021
Generative Scene Networks is introduced, which learns to decompose scenes into a collection of many local radiance fields that can be rendered from a free moving camera, and which produces quantitatively higher-quality scene renderings across several different scene datasets. Expand
pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis
• Computer Science
• CVPR
• 2021
This work proposes a novel generative model, named Periodic Implicit Generative Adversarial Networks ($\pi$-GAN or pi-GAN), for high-quality 3D-aware image synthesis that leverages neural representations with periodic activation functions and volumetric rendering to represent scenes as view-consistent 3D representations with fine detail. Expand
Editing Conditional Radiance Fields
This paper introduces a method for propagating coarse 2D user scribbles to the 3D space, to modify the color or shape of a local region, and proposes a conditional radiance field that incorporates new modular network components, including a shape branch that is shared across object instances. Expand
Mixture of volumetric primitives for efficient neural rendering
• Computer Science
• ArXiv
• 2021
Mixture of Volumetric Primitives (MVP), a representation for rendering dynamic 3D content that combines the completeness of volumetric representations with the efficiency of primitive-based rendering, is presented. Expand
RGB-D Local Implicit Function for Depth Completion of Transparent Objects
A new approach for depth completion of transparent objects from a single RGB-D image using a local implicit neural representation built on ray-voxel pairs that allows the method to generalize to unseen objects and achieve fast inference speed. Expand
StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis
• Jiatao Gu, Lingjie Liu, Peng Wang
• Computer Science, Mathematics
• ArXiv
• 2021
We propose StyleNeRF, a 3D-aware generative model for photo-realistic highresolution image synthesis with high multi-view consistency, which can be trained on unstructured 2D images. ExistingExpand
3D Scene Compression through Entropy Penalized Neural Representation Functions
• Computer Science, Engineering
• 2021 Picture Coding Symposium (PCS)
• 2021
This work significantly outperforms a state-of-the-art conventional approach for scene compression, achieving simultaneously higher quality reconstructions and lower bitrates, and shows that the performance at lower bitrate can be improved by jointly representing multiple scenes using a soft form of parameter sharing. Expand
A-NeRF: Surface-free Human 3D Pose Refinement via Neural Rendering
• Computer Science
• ArXiv
• 2021
Fig. 1. Our A-NeRF test-time optimization for monocular 3D human pose estimation jointly learns a volumetric body model of the user that can be animated and works with diverse body shapes (left),Expand

#### References

SHOWING 1-10 OF 85 REFERENCES
Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis
• Computer Science
• 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 2020
This work defines the new task of 3D controllable image synthesis and proposes an approach for solving it by reasoning both in 3D space and in the 2D image domain, and demonstrates that the model is able to disentangle latent 3D factors of simple multi-object scenes in an unsupervised fashion from raw images. Expand
Visual Object Networks: Image Generation with Disentangled 3D Representations
A new generative model, Visual Object Networks (VONs), synthesizing natural images of objects with a disentangled 3D representation that enables many 3D operations such as changing the viewpoint of a generated image, shape and texture editing, linear interpolation in texture and shape space, and transferring appearance across different objects and viewpoints. Expand
HoloGAN: Unsupervised Learning of 3D Representations From Natural Images
• Computer Science
• 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
• 2019
HoloGAN is the first generative model that learns 3D representations from natural images in an entirely unsupervised manner and is shown to be able to generate images with similar or higher visual quality than other generative models. Expand
Monocular Neural Image Based Rendering With Continuous View Control
• Jie Song
• Computer Science
• 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
• 2019
The experiments show that both proposed components, the transforming encoder-decoder and depth-guided appearance mapping, lead to significantly improved generalization beyond the training views and in consequence to more accurate view synthesis under continuous 6-DoF camera control. Expand
Stereo Magnification: Learning View Synthesis using Multiplane Images
• Computer Science
• ArXiv
• 2018
This paper explores an intriguing scenario for view synthesis: extrapolating views from imagery captured by narrow-baseline stereo cameras, including VR cameras and now-widespread dual-lens camera phones, and proposes a learning framework that leverages a new layered representation that is called multiplane images (MPIs). Expand
Geometric Image Synthesis
• Computer Science
• ACCV
• 2018
This work proposes a trainable, geometry-aware image generation method that leverages various types of scene information, including geometry and segmentation, to create realistic looking natural images that match the desired scene structure. Expand
DeepVoxels: Learning Persistent 3D Feature Embeddings
• Computer Science
• 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 2019
This work proposes DeepVoxels, a learned representation that encodes the view-dependent appearance of a 3D scene without having to explicitly model its geometry, based on a Cartesian 3D grid of persistent embedded features that learn to make use of the underlying3D scene structure. Expand
Pushing the Boundaries of View Extrapolation With Multiplane Images
• Computer Science
• 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 2019
This paper presents a theoretical analysis showing how the range of views that can be rendered from an MPI increases linearly with the MPI disparity sampling frequency, as well as a novel MPI prediction procedure that theoretically enables view extrapolations of up to 4 times the lateral viewpoint movement allowed by prior work. Expand
Learning Implicit Surface Light Fields
• Computer Science
• 2020 International Conference on 3D Vision (3DV)
• 2020
This work proposes a novel implicit representation for capturing the visual appearance of an object in terms of its surface light field and shows that the proposed representation can be embedded into a variational auto-encoder for generating novel appearances that conform to the specified illumination conditions. Expand
Pix2Vox: Context-Aware 3D Reconstruction From Single and Multi-View Images
• Computer Science
• 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
• 2019
Experimental results on the ShapeNet and Pix3D benchmarks indicate that the proposed Pix2Vox outperforms state-of-the-arts by a large margin, and the proposed method is 24 times faster than 3D-R2N2 in terms of backward inference time. Expand