RTMV: A Ray-Traced Multi-View Synthetic Dataset for Novel View Synthesis

  title={RTMV: A Ray-Traced Multi-View Synthetic Dataset for Novel View Synthesis},
  author={Jonathan Tremblay and Moustafa Meshry and Alex Evans and Jan Kautz and Alexander Keller and S. Khamis and Charles T. Loop and Nate Morrical and Koki Nagano and Towaki Takikawa and Stan Birchfield},
We present a large-scale synthetic dataset for novel view synthesis consisting of ∼ 300k images rendered from nearly 2000 complex scenes using high-quality ray tracing at high resolution ( 1600 × 1600 pixels). The dataset is orders of magnitude larger than existing synthetic datasets for novel view synthesis, thus providing a large unified benchmark for both training and evaluation. Using 4 distinct sources of high-quality 3D meshes, the scenes of our dataset exhibit challenging variations in… 

Variable Bitrate Neural Fields

A dictionary method for compressing feature grids, reducing their memory consumption by up to 100 × and permitting a multiresolution representation which can be useful for out-of-core streaming is presented.



IBRNet: Learning Multi-View Image-Based Rendering

A method that synthesizes novel views of complex scenes by interpolating a sparse set of nearby views using a network architecture that includes a multilayer perceptron and a ray transformer that estimates radiance and volume density at continuous 5D locations.

Neural Sparse Voxel Fields

This work introduces Neural Sparse Voxel Fields (NSVF), a new neural scene representation for fast and high-quality free-viewpoint rendering that is over 10 times faster than the state-of-the-art (namely, NeRF) at inference time while achieving higher quality results.

BlendedMVS: A Large-Scale Dataset for Generalized Multi-View Stereo Networks

  • Yao YaoZixin Luo Long Quan
  • Computer Science
    2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2020
This paper introduces BlendedMVS, a novel large-scale dataset to provide sufficient training ground truth for learning-based MVS and endows the trained model with significantly better generalization ability compared with other MVS datasets.

NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis

This work describes how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrates results that outperform prior work on neural rendering and view synthesis.

Free View Synthesis

This work presents a method for novel view synthesis from input images that are freely distributed around a scene that can synthesize images for free camera movement through the scene, and works for general scenes with unconstrained geometric layouts.

GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis

This paper proposes a generative model for radiance fields which have recently proven successful for novel view synthesis of a single scene, and introduces a multi-scale patch-based discriminator to demonstrate synthesis of high-resolution images while training the model from unposed 2D images alone.

MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo

This work proposes a generic deep neural network that can reconstruct radiance fields from only three nearby input views via fast network inference, and leverages plane-swept cost volumes for geometry-aware scene reasoning, and combines this with physically based volume rendering for neural radiance field reconstruction.

Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines

An algorithm for view synthesis from an irregular grid of sampled views that first expands each sampled view into a local light field via a multiplane image (MPI) scene representation, then renders novel views by blending adjacent local light fields.

Stereo Magnification: Learning View Synthesis using Multiplane Images

This paper explores an intriguing scenario for view synthesis: extrapolating views from imagery captured by narrow-baseline stereo cameras, including VR cameras and now-widespread dual-lens camera phones, and proposes a learning framework that leverages a new layered representation that is called multiplane images (MPIs).

Silhouette‐Aware Warping for Image‐Based Rendering

This work formulate silhouette‐aware warps that preserve salient depth discontinuities and improves the rendering of difficult foreground objects, even when deviating from view interpolation, which results in good quality IBR for previously challenging environments.