Learning to Synthesize a 4D RGBD Light Field from a Single Image

@article{Srinivasan2017LearningTS,
  title={Learning to Synthesize a 4D RGBD Light Field from a Single Image},
  author={Pratul P. Srinivasan and Tongzhou Wang and Ashwin Sreelal and Ravi Ramamoorthi and Ren Ng},
  journal={2017 IEEE International Conference on Computer Vision (ICCV)},
  year={2017},
  pages={2262-2270}
}
We present a machine learning algorithm that takes as input a 2D RGB image and synthesizes a 4D RGBD light field (color and depth of the scene in each ray direction. [] Key Method Our synthesis pipeline consists of a convolutional neural network (CNN) that estimates scene geometry, a stage that renders a Lambertian light field using that geometry, and a second CNN that predicts occluded rays and non-Lambertian effects. Our algorithm builds on recent view synthesis methods, but is unique in predicting RGBD…
Synthesizing a 4D Spatio-Angular Consistent Light Field from a Single Image
TLDR
A single end-to-end deep neural network that does not require a physical-based approach nor a post-processing subnetwork is proposed and successfully synthesizes dense light fields and qualitatively and quantitatively outperforms the previous model.
Synthesizing light field from a single image with variable MPI and two network fusion
TLDR
This work extends the multiplane image (MPI) representation by allowing the disparity of the layers to be inferred from the input image, and shows that, compared to the original MPI representation, this representation models the scenes more accurately.
Learning Light Field Synthesis with Multi-Plane Images: Scene Encoding as a Recurrent Segmentation Task
TLDR
This paper addresses the problem of view synthesis from large baseline light fields, by turning a sparse set of input views into a Multi-plane Image (MPI), and proposes a lightweight network that does not require extensive training.
Light-Field Raw Data Synthesis From RGB-D Images: Pushing to the Extreme
TLDR
An all-software algorithm to synthesize light-field raw data from a single RGB-D input image, which is driven largely by the need in the research area of light- field data compression.
Acquiring Dynamic Light Fields Through Coded Aperture Camera
TLDR
This study proposes a method for acquiring a dynamic light field (a moving scene) using a coded aperture camera and a convolutional neural network (CNN), and develops a new configuration of image observation, called V-shape observation, and trains the CNN using a dynamic-light-field dataset with pseudo motions.
EPINET: A Fully-Convolutional Neural Network Using Epipolar Geometry for Depth from Light Field Images
TLDR
This paper introduces a fast and accurate light field depth estimation method based on a fully-convolutional neural network that achieved the top rank in the HCI 4D Light Field Benchmark on most metrics, and demonstrates the effectiveness of the proposed method on real-world light-field images.
Learning to Capture Light Fields Through a Coded Aperture Camera
TLDR
A learning-based framework for acquiring a light field through a coded aperture camera with superior performance over several state-of-the-art methods and can successfully learn good image-acquisition and reconstruction strategies.
5D Light Field Synthesis from a Monocular Video
TLDR
Experimental results show that the model can successfully synthesize the light field video for synthetic and real scenes and outperforms the previous frame-by-frame method quantitatively and qualitatively.
Using a Depth Heuristic for Light Field Volume Rendering
TLDR
The application of depth heuristics is a novel contribution to light field volume rendering, leading to high quality view synthesis which is further improved by a Convolutional Neural Network.
...
...

References

SHOWING 1-10 OF 41 REFERENCES
Learning-based view synthesis for light field cameras
TLDR
This paper proposes a novel learning-based approach to synthesize new views from a sparse set of input views that could potentially decrease the required angular resolution of consumer light field cameras, which allows their spatial resolution to increase.
Multi-view 3D Models from Single Images with a Convolutional Network
TLDR
A convolutional network capable of inferring a 3D representation of a previously unseen object given a single image of this object and several depth maps fused together give a full point cloud of the object.
View Synthesis by Appearance Flow
TLDR
This work addresses the problem of novel view synthesis: given an input image, synthesizing new images of the same object or scene observed from arbitrary viewpoints and shows that for both objects and scenes, this approach is able to synthesize novel views of higher perceptual quality than previous CNN-based techniques.
Oriented Light-Field Windows for Scene Flow
TLDR
This paper proposes an oriented 4D light-field window that accounts for shearing, translation, and windowing in scene flow, a generalization of optical flow to the 3D vector field describing the motion of each point in the scene.
Depth synthesis and local warps for plausible image-based navigation
TLDR
This work introduces a new IBR algorithm that is robust to missing or unreliable geometry, providing plausible novel views even in regions quite far from the input camera positions, and demonstrates novel view synthesis in real time for multiple challenging scenes with significant depth complexity.
Light field rendering
TLDR
This paper describes a sampled representation for light fields that allows for both efficient creation and display of inward and outward looking views, and describes a compression system that is able to compress the light fields generated by more than a factor of 100:1 with very little loss of fidelity.
Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
TLDR
This work proposes a unsupervised framework to learn a deep convolutional neural network for single view depth prediction, without requiring a pre-training stage or annotated ground-truth depths, and shows that this network trained on less than half of the KITTI dataset gives comparable performance to that of the state-of-the-art supervised methods for singleView depth estimation.
Accurate depth map estimation from a lenslet light field camera
TLDR
This paper introduces an algorithm that accurately estimates depth maps using a lenslet light field camera and estimates the multi-view stereo correspondences with sub-pixel accuracy using the cost volume using the phase shift theorem.
Unsupervised Monocular Depth Estimation with Left-Right Consistency
TLDR
This paper proposes a novel training objective that enables the convolutional neural network to learn to perform single image depth estimation, despite the absence of ground truth depth data, and produces state of the art results for monocular depth estimation on the KITTI driving dataset.
Light field from micro-baseline image pair
TLDR
A Disparity Assisted Phase based Synthesis strategy that can integrate disparity information into the phase term of a reference image to warp it to its close neighbor views to solve the problems of disparity inconsistency and ringing artifact in available phase-based view synthesis methods is introduced.
...
...