Depth-supervised NeRF: Fewer Views and Faster Training for Free

@article{Deng2022DepthsupervisedNF,
  title={Depth-supervised NeRF: Fewer Views and Faster Training for Free},
  author={Kangle Deng and Andrew Liu and Junyan Zhu and Deva Ramanan},
  journal={2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2022},
  pages={12872-12881}
}
  • Kangle Deng, Andrew Liu, D. Ramanan
  • Published 6 July 2021
  • Computer Science
  • 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
A commonly observed failure mode of Neural Radiance Field (NeRF) is fitting incorrect geometries when given an insufficient number of input views. One potential reason is that standard volumetric rendering does not enforce the constraint that most of a scene's geometry consist of empty space and opaque surfaces. We formalize the above assumption through DS-NeRF (Depth-supervised Neural Radiance Fields), a loss for learning radiance fields that takes advantage of readily-available depth… 

NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fields

TLDR
This paper demonstrates that a NeRF representation of a scene can be used to train dense object descriptors and demonstrates the learned dense descriptors enable robots to perform accurate 6-degree of freedom (6-DoF) pick and place of thin and reflective objects.

SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Image

TLDR
SinNeRF constructs a semi-supervised learning process, where it introduces and propagate geometry pseudo labels and semantic pseudo labels to guide the progressive training process, and shows that even without pre-training on multi-view datasets, SinNeRF can yield photo-realistic novel-view synthesis results.

Mip-NeRF RGB-D: Depth Assisted Fast Neural Radiance Fields

TLDR
The recently proposed Mip-NeRF approach, which uses conical frustums instead of rays for volume rendering, allows to address major limitations of NeRF-based approaches including improving the accuracy of geometry, reduced artifacts, faster training time, and shortened prediction time.

Dense Depth Priors for Neural Radiance Fields from Sparse Input Views

TLDR
This work uses sparse depth data that is freely available from the structure from motion (SfM) preprocessing step used to estimate camera poses to convert these sparse points into dense depth maps and uncertainty estimates, which are used to guide NeRF optimization.

RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs

TLDR
This work ad-ditionally uses a normalizing flow model to regularize the color of unobserved viewpoints in NeRF, and outperforms not only other methods that optimize over a single scene, but in many cases also conditional models that are extensively pre-trained on large multi-view datasets.

DoF-NeRF: Depth-of-Field Meets Neural Radiance Fields

TLDR
DoF-NeRF is introduced, a novel neural rendering approach that can deal with shallow DoF inputs and can simulate DoF effect and extends NeRF to simulate the aperture of lens following the principles of geometric optics.

DDNeRF: Depth Distribution Neural Radiance Fields

TLDR
This work presents depth distribution neural radiance field (DDNeRF), a new method that significantly increases sampling efficiency along rays during training while achieving superior results for a given sampling budget by learning a more accurate representation of the density distribution along rays.

NeRF, meet differential geometry!

TLDR
This work shows how a direct mathematical formalism of previously proposed NeRF variants aimed at improving the performance in challenging conditions can be used to natively encourage the regularity of surfaces (by means of Gaussian and Mean Curvatures) making it possible, for example, to learn surfaces from a very limited number of views.

Gaussian Activated Neural Radiance Fields for High Fidelity Reconstruction & Pose Estimation

TLDR
Gaussian Activated neural Radiance Fields (GARF) is presented as a new positional embedding-free neural radiance field architecture – employing Gaussian activations – that outperforms the current state-of-the-art in terms of high fidelity reconstruction and pose estimation.

Beyond RGB: Scene-Property Synthesis with Neural Radiance Fields

TLDR
This paper provides a new approach to scene understanding, from a synthesis model perspective, by leveraging the recent progress on implicit 3D representation and neural rendering by introducing SceneProperty Synthesis with NeRF (SS-NeRF), a powerful tool for bridging generative learning and discriminative learning.
...

References

SHOWING 1-10 OF 50 REFERENCES

NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis

TLDR
This work describes how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrates results that outperform prior work on neural rendering and view synthesis.

Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis

TLDR
DietNeRF improves the perceptual quality of few-shot view synthesis when learned from scratch, can render novel views with as few as one observed image when pre-trained on a multi-view dataset, and produces plausible completions of completely unobserved regions.

NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections

TLDR
A learning-based method for synthesizing novel views of complex scenes using only unstructured collections of in-the-wild photographs, and applies it to internet photo collections of famous landmarks, to demonstrate temporally consistent novel view renderings that are significantly closer to photorealism than the prior state of the art.

MegaDepth: Learning Single-View Depth Prediction from Internet Photos

  • Zhengqi LiNoah Snavely
  • Computer Science
    2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
  • 2018
TLDR
This work proposes to use multi-View Internet photo collections, a virtually unlimited data source, to generate training data via modern structure-from-motion and multi-view stereo (MVS) methods, and presents a large depth dataset called MegaDepth based on this idea.

NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo

TLDR
A new multi-view depth estimation method that utilizes both conventional SfM reconstruction and learning-based priors over the recently proposed neural radiance fields (NeRF), with surprising findings presented on the effectiveness of correspondence-based opti-mization and NeRF-based optimization over the adapted depth priors.

iNeRF: Inverting Neural Radiance Fields for Pose Estimation

TLDR
iNeRF can perform categorylevel object pose estimation, including object instances not seen during training, with RGB images by inverting a NeRF model inferred from a single view.

Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenes

TLDR
Stereo Radiance Fields is introduced, a neural view synthesis approach that is trained end-to-end, generalizes to new scenes, and requires only sparse views at test time, andExperiments show that SRF learns structure instead of over-fitting on a scene, achieving significantly sharper, more detailed results than scene-specific models.

pixelNeRF: Neural Radiance Fields from One or Few Images

We propose pixelNeRF, a learning framework that predicts a continuous neural scene representation conditioned on one or few input images. The existing approach for constructing neural radiance fields

DONeRF: Towards Real‐Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks

TLDR
DONeRF, a compact dual network design with a depth oracle network as its first step and a locally sampled shading network for ray accumulation, is presented, which reduces the inference costs by up to 48× compared to NeRF when conditioning on available ground truth depth information.

IBRNet: Learning Multi-View Image-Based Rendering

TLDR
A method that synthesizes novel views of complex scenes by interpolating a sparse set of nearby views using a network architecture that includes a multilayer perceptron and a ray transformer that estimates radiance and volume density at continuous 5D locations.