Crowdsampling the Plenoptic Function

@inproceedings{Li2020CrowdsamplingTP,
  title={Crowdsampling the Plenoptic Function},
  author={Zhengqi Li and Wenqi Xian and Abe Davis and Noah Snavely},
  booktitle={ECCV},
  year={2020}
}
Many popular tourist landmarks are captured in a multitude of online, public photos. These photos represent a sparse and unstructured sampling of the plenoptic function for a particular scene. In this paper,we present a new approach to novel view synthesis under time-varying illumination from such data. Our approach builds on the recent multi-plane image (MPI) format for representing local light fields under fixed viewing conditions. We introduce a new DeepMPI representation, motivated by… Expand
FreeStyleGAN: Free-view Editable Portrait Rendering with the Camera Manifold
Fig. 1. We introduce a new approach that generates an image with StyleGAN defined by a precise 3D camera. This enables faces synthesized with StyleGAN to be used in 3D free-viewpoint rendering, whileExpand
Stable View Synthesis
TLDR
Experimental results demonstrate that SVS outperforms state-of-the-art view synthesis methods both quantitatively and qualitatively on three diverse real-world datasets, achieving unprecedented levels of realism in free-viewpoint video of challenging large-scale scenes. Expand
Vid2Actor: Free-viewpoint Animatable Person Synthesis from Video in the Wild
Given an “in-the-wild” video of a person, we reconstruct an animatable model of the person in the video. The output model can be rendered in any body pose to any camera view, via the learnedExpand
Moving in a 360 World: Synthesizing Panoramic Parallaxes from a Single Panorama
TLDR
OmniNeRF is presented, the first method to the application of parallaxenabled novel panoramic view synthesis, and an Omnidirectional Neural Radiance Field with visible pixels collecting from omnid Directional viewing angles at a fixed center for the estimation of new viewing angles from varying camera positions is optimized. Expand
Geometry-Free View Synthesis: Transformers and no 3D Priors
TLDR
The experiments show that no such geometric priors are required and that the transformer is capable of implicitly learning 3D relationships between images, and this approach outperforms the state of the art in terms of visual quality while covering the full distribution of possible realizations. Expand
Towers of Babel: Combining Images, Language, and 3D Geometry for Learning Multimodal Vision
TLDR
This work presents WikiScenes, a new, large-scale dataset of landmark photo collections that contains descriptive text in the form of captions and hierarchical category names that forms a new testbed for multimodal reasoning involving images, text, and 3D geometry. Expand
Editing Conditional Radiance Fields
TLDR
This paper introduces a method for propagating coarse 2D user scribbles to the 3D space, to modify the color or shape of a local region, and proposes a conditional radiance field that incorporates new modular network components, including a shape branch that is shared across object instances. Expand
PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting
TLDR
PhySG is presented, an end-to-end inverse rendering pipeline that includes a fully differentiable renderer, and can reconstruct geometry, materials, and illumination from scratch from a set of images, and demonstrates, with both synthetic and real data, that it can enable rendering of novel viewpoints, but also physics-based appearance editing of materials and illumination. Expand
NeuralPlan: Neural floorplan radiance fields for accelerated view synthesis
TLDR
This approach performs significantly accelerated learning of neural floorplan radiance fields in around 15 min for full buildings on a single commodity GPU, and renders in real-time at 64 Hz, allowing for immersive visual experiences. Expand
Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies
TLDR
Based on the skeleton-driven deformation, blend weight fields are used with 3D human skeletons to generate observation-tocanonical and canonical-to-observation correspondences and can regularize the learning of deformation fields. Expand
...
1
2
3
...

References

SHOWING 1-10 OF 81 REFERENCES
Multi-view relighting using a geometry-aware network
TLDR
This work proposes the first learning-based algorithm that can relight images in a plausible and controllable manner given multiple views of an outdoor scene using a geometry-aware neural network that utilizes multiple geometry cues and source and target shadow masks computed from a noisy proxy geometry obtained by multi-view stereo. Expand
Deep view synthesis from sparse photometric images
TLDR
This paper synthesizes novel viewpoints across a wide range of viewing directions (covering a 60° cone) from a sparse set of just six viewing directions, based on a deep convolutional network trained to directly synthesize new views from the six input views. Expand
Stereo magnification
TLDR
This paper explores an intriguing scenario for view synthesis: extrapolating views from imagery captured by narrow-baseline stereo cameras, including VR cameras and now-widespread dual-lens camera phones, and proposes a learning framework that leverages a new layered representation that is called multiplane images (MPIs). Expand
Neural Rerendering in the Wild
TLDR
This work applies traditional 3D reconstruction to register the photos and approximate the scene as a point cloud from Internet photos of a tourist landmark, and trains a deep neural network to learn the mapping of these initial renderings to the actual photos. Expand
Depth synthesis and local warps for plausible image-based navigation
TLDR
This work introduces a new IBR algorithm that is robust to missing or unreliable geometry, providing plausible novel views even in regions quite far from the input camera positions, and demonstrates novel view synthesis in real time for multiple challenging scenes with significant depth complexity. Expand
Local light field fusion
TLDR
An algorithm for view synthesis from an irregular grid of sampled views that first expands each sampled view into a local light field via a multiplane image (MPI) scene representation, then renders novel views by blending adjacent local light fields. Expand
Coherent intrinsic images from photo collections
TLDR
This paper uses multi-view stereo to automatically reconstruct 3D points and normals from which it derives relationships between reflectance values at different locations, across multiple views and consequently different lighting conditions, and uses robust estimation to reliably identify reflectance ratios between pairs of points. Expand
Learning-based view synthesis for light field cameras
TLDR
This paper proposes a novel learning-based approach to synthesize new views from a sparse set of input views that could potentially decrease the required angular resolution of consumer light field cameras, which allows their spatial resolution to increase. Expand
Casual 3D photography
TLDR
An algorithm that enables casual 3D photography and proposes a novel parallax-tolerant stitching algorithm that warps the depth maps into the central panorama and stitches two color-and-depth panoramas for the front and back scene surfaces. Expand
Scribbler: Controlling Deep Image Synthesis with Sketch and Color
TLDR
A deep adversarial image synthesis architecture that is conditioned on sketched boundaries and sparse color strokes to generate realistic cars, bedrooms, or faces is proposed and demonstrates a sketch based image synthesis system which allows users to scribble over the sketch to indicate preferred color for objects. Expand
...
1
2
3
4
5
...