RTMV: A Ray-Traced Multi-View Synthetic Dataset for Novel View Synthesis

@article{Tremblay2022RTMVAR,
  title={RTMV: A Ray-Traced Multi-View Synthetic Dataset for Novel View Synthesis},
  author={Jonathan Tremblay and Moustafa Meshry and Alex Evans and Jan Kautz and Alexander Keller and S. Khamis and Charles T. Loop and Nate Morrical and Koki Nagano and Towaki Takikawa and Stan Birchfield},
  journal={ArXiv},
  year={2022},
  volume={abs/2205.07058}
}
We present a large-scale synthetic dataset for novel view synthesis consisting of ∼ 300k images rendered from nearly 2000 complex scenes using high-quality ray tracing at high resolution ( 1600 × 1600 pixels). The dataset is orders of magnitude larger than existing synthetic datasets for novel view synthesis, thus providing a large unified benchmark for both training and evaluation. Using 4 distinct sources of high-quality 3D meshes, the scenes of our dataset exhibit challenging variations in… 

Variable Bitrate Neural Fields

TLDR
A dictionary method for compressing feature grids, reducing their memory consumption by up to 100 × and permitting a multiresolution representation which can be useful for out-of-core streaming is presented.

References

SHOWING 1-10 OF 101 REFERENCES

IBRNet: Learning Multi-View Image-Based Rendering

TLDR
A method that synthesizes novel views of complex scenes by interpolating a sparse set of nearby views using a network architecture that includes a multilayer perceptron and a ray transformer that estimates radiance and volume density at continuous 5D locations.

Neural Sparse Voxel Fields

TLDR
This work introduces Neural Sparse Voxel Fields (NSVF), a new neural scene representation for fast and high-quality free-viewpoint rendering that is over 10 times faster than the state-of-the-art (namely, NeRF) at inference time while achieving higher quality results.

BlendedMVS: A Large-Scale Dataset for Generalized Multi-View Stereo Networks

  • Yao YaoZixin Luo Long Quan
  • Computer Science
    2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2020
TLDR
This paper introduces BlendedMVS, a novel large-scale dataset to provide sufficient training ground truth for learning-based MVS and endows the trained model with significantly better generalization ability compared with other MVS datasets.

NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis

TLDR
This work describes how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrates results that outperform prior work on neural rendering and view synthesis.

Free View Synthesis

TLDR
This work presents a method for novel view synthesis from input images that are freely distributed around a scene that can synthesize images for free camera movement through the scene, and works for general scenes with unconstrained geometric layouts.

GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis

TLDR
This paper proposes a generative model for radiance fields which have recently proven successful for novel view synthesis of a single scene, and introduces a multi-scale patch-based discriminator to demonstrate synthesis of high-resolution images while training the model from unposed 2D images alone.

MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo

TLDR
This work proposes a generic deep neural network that can reconstruct radiance fields from only three nearby input views via fast network inference, and leverages plane-swept cost volumes for geometry-aware scene reasoning, and combines this with physically based volume rendering for neural radiance field reconstruction.

Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines

TLDR
An algorithm for view synthesis from an irregular grid of sampled views that first expands each sampled view into a local light field via a multiplane image (MPI) scene representation, then renders novel views by blending adjacent local light fields.

Stereo Magnification: Learning View Synthesis using Multiplane Images

TLDR
This paper explores an intriguing scenario for view synthesis: extrapolating views from imagery captured by narrow-baseline stereo cameras, including VR cameras and now-widespread dual-lens camera phones, and proposes a learning framework that leverages a new layered representation that is called multiplane images (MPIs).

Silhouette‐Aware Warping for Image‐Based Rendering

TLDR
This work formulate silhouette‐aware warps that preserve salient depth discontinuities and improves the rendering of difficult foreground objects, even when deviating from view interpolation, which results in good quality IBR for previously challenging environments.
...