Nerfies: Deformable Neural Radiance Fields

@article{Park2021NerfiesDN,
  title={Nerfies: Deformable Neural Radiance Fields},
  author={Keunhong Park and U. Sinha and Jonathan T. Barron and Sofien Bouaziz and Dan B. Goldman and Steven M. Seitz and Ricardo Martin-Brualla},
  journal={2021 IEEE/CVF International Conference on Computer Vision (ICCV)},
  year={2021},
  pages={5845-5854}
}
We present the first method capable of photorealistically reconstructing deformable scenes using photos/videos captured casually from mobile phones. Our approach augments neural radiance fields (NeRF) by optimizing an additional continuous volumetric deformation field that warps each observed point into a canonical 5D NeRF. We observe that these NeRF-like deformation fields are prone to local minima, and propose a coarse-to-fine optimization method for coordinate-based models that allows for… 
Disentangled3D: Learning a 3D Generative Model with Disentangled Geometry and Appearance from Monocular Images
TLDR
This paper designs a 3D GAN which can learn a disentangled model of objects, just from monocular observations, and designs an approach to embed real images into the latent space of the model, enabling editing of real images.
H-NeRF: Neural Radiance Fields for Rendering and Temporal Reconstruction of Humans in Motion
TLDR
This approach combines ideas from neural scene representation, novel-view synthesis, and implicit statistical geometric human representations, coupled using novel loss functions to constrain a radiance field by a structured implicit human body model, represented using signed distance functions.
Neural Articulated Radiance Field
TLDR
Neural Articulated Radiance Field is a novel deformable 3D representation for articulated objects learned from images that is fully differentiable and can be trained from images with pose annotations through the use of an autoencoder.
Point-NeRF: Point-based Neural Radiance Fields
TLDR
Point-NeRF can be combined with other 3D reconstruction methods and handles the errors and outliers in such methods via a novel pruning and growing mechanism and achieve the state-of-the-art results.
Real-Time Neural Character Rendering with Pose-Guided Multiplane Images
TLDR
This work proposes pose-guided multiplane image (MPI) synthesis which can render an animatable character in real scenes with photorealistic quality and demonstrates advanta-geous novel-view synthesis quality over the state-of-the-art approaches for characters with challenging motions.
BANMo: Building Animatable 3D Neural Models from Many Casual Videos
TLDR
The key insight is to merge three schools of thought; classic deformable shape models that make use of articulated bones and blend skinning, volumetric neural radiance fields (NeRFs) that are amenable to gradient-based optimization, and canonical embeddings that generate correspondences between pixels and an articulated model.
CityNeRF: Building NeRF at City Scale
TLDR
This work makes the first attempt to bring NeRF to city-scale, with views ranging from satellite-level that captures the overview of a city, to ground-level imagery showing complex details of an architecture.
I M Avatar: Implicit Morphable Head Avatars from Videos
TLDR
This work proposes IMavatar, a novel method for learning implicit head avatars from monocular videos that improves geometry and covers a more complete expression space compared to state-of-the-art methods.
Neural Articulated Radiance Field Supplemental Material
TLDR
This work quantitatively evaluates the model in the case of a single articulated 3D object and supplements those results with the quantitative ablation study for Autoencoder in Table A.
HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields
TLDR
By modeling a family of shapes in a high dimensional space, the Hyper-NeRF model is able to handle topological variation and thereby produce more realistic renderings and more accurate geometric reconstructions, as can be seen in (b).
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 73 REFERENCES
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
TLDR
A new dataset of human perceptual similarity judgments is introduced and it is found that deep features outperform all previous metrics by large margins on this dataset, and suggests that perceptual similarity is an emergent property shared across deep visual representations.
Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes
TLDR
A method to perform novel view and time synthesis of dynamic scenes, requiring only a monocular video with known camera poses as input, is presented, and a new representation that models the dynamic scene as a time-variant continuous function of appearance, geometry, and 3D scene motion is introduced.
Novel View Synthesis of Dynamic Scenes With Globally Coherent Depths From a Monocular Camera
TLDR
This paper presents a new method to synthesize an image from arbitrary views and times given a collection of images of a dynamic scene, and evaluates the method of depth estimation and view synthesis on a diverse real-world dynamic scenes and shows the outstanding performance over existing methods.
Neural volumes: learning dynamic renderable volumes from images
  • ACM ToG,
  • 2019
NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections
TLDR
A learning-based method for synthesizing novel views of complex scenes using only unstructured collections of in-the-wild photographs, and applies it to internet photo collections of famous landmarks, to demonstrate temporally consistent novel view renderings that are significantly closer to photorealism than the prior state of the art.
Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
TLDR
An approach for selecting problem-specific Fourier features that greatly improves the performance of MLPs for low-dimensional regression tasks relevant to the computer vision and graphics communities is suggested.
Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer
TLDR
This work proposes a robust training objective that is invariant to changes in depth range and scale, advocate the use of principled multi-objective learning to combine data from different sources, and highlights the importance of pretraining encoders on auxiliary tasks.
D-NeRF: Neural Radiance Fields for Dynamic Scenes
TLDR
D-NeRF is introduced, a method that extends neural radiance fields to a dynamic domain, allowing to reconstruct and render novel images of objects under rigid and non-rigid motions from a single camera moving around the scene.
Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video
TLDR
Non-Rigid Neural Radiance Fields (NR-NeRF), a reconstruction and novel view synthesis approach for general non-rigid dynamic scenes, takes RGB images of a dynamic scene as input, and creates a high-quality space-time geometry and appearance representation.
Nonrigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video, 2021
  • 2021
...
1
2
3
4
5
...