Fast View Synthesis with Deep Stereo Vision

  title={Fast View Synthesis with Deep Stereo Vision},
  author={Tewodros Habtegebrial and K. Varanasi and Christian Bailer and D. Stricker},
Novel view synthesis is an important problem in computer vision and graphics. [...] Key Method Both tasks are structured prediction problems that could be effectively learned with CNNs. Experiments on the KITTI Odometry dataset show that our approach is more accurate and significantly faster than the current state-of-the-art. The code and supplementary material will be publicly available. Results could be found here this https URLExpand
Deep Learning based Novel View Synthesis
A deep convolutional neural network (CNN) which learns to predict novel views of a scene from given collection of images and estimates, at each pixel, the probability distribution over possible depth levels in the scene. Expand
Deep View Synthesis via Self-Consistent Generative Network
A novel deep generative model, called Self-Consistent Generative Network (SCGN), which synthesizes novel views from the given input views without explicitly exploiting the geometric information, and adversarial loss is introduced to improve the photo-realism of novel views. Expand
Image-Based Rendering for Large-Scale Outdoor Scenes With Fusion of Monocular and Multi-View Stereo Depth
A novel depth refinement method that combines MVS depth maps with monocular depth maps predicted via deep learning and an efficient shape-preserving warping algorithm that uses superpixels to generate the warped images and blend expected novel views of scenes is proposed. Expand
A Lightweight Neural Network for Monocular View Generation With Occlusion Handling
  • Simon Evain, C. Guillemot
  • Computer Science, Medicine
  • IEEE Transactions on Pattern Analysis and Machine Intelligence
  • 2021
A very lightweight neural network architecture, trained on stereo data pairs, which performs view synthesis from one single image, which outperforms visually and metric-wise state-of-the-art approaches on the challenging KITTI dataset, all while reducing by a very significant order of magnitude the required number of parameters. Expand
Depth-guided learning light field angular super-resolution with edge-aware inpainting
Experimental results show that the proposed end-to-end learning-based approach to achieve angular super-resolution of the light field with a wide-baseline outperforms the state-of-the-art light field angularsuper-resolution methods, especially in the terms of visual performance near edges. Expand
With the emergence of deep neural networks , learning-based techniques have become an increasingly popular tool for novel view synthesis
Fig. 1. 3D Ken Burns effect from a single image. Given a single input image and optional user annotations in form of two cropping windows, our framework animates the input image while adding parallaxExpand
Predicting Novel Views Using Generative Adversarial Query Network
The proposed GAQN is a general learning framework for novel view synthesis that combines Generative Query Network (GQN) and Generative Adversarial Networks (GANs) and introduces a feature-matching loss function for stabilizing the training procedure. Expand
3D Ken Burns effect from a single image
A framework that synthesizes the 3D Ken Burns effect from a single image, supporting both a fully automatic mode and an interactive mode with the user controlling the camera is introduced, and demonstrates that the system allows users to achieve better results while requiring little effort compared to existing solutions. Expand


View Synthesis by Appearance Flow
This work addresses the problem of novel view synthesis: given an input image, synthesizing new images of the same object or scene observed from arbitrary viewpoints and shows that for both objects and scenes, this approach is able to synthesize novel views of higher perceptual quality than previous CNN-based techniques. Expand
Geometry-Aware Deep Network for Single-Image Novel View Synthesis
A new region-aware geometric transform network is developed that performs these multiple tasks in a common framework and demonstrates the effectiveness of the network in generating high-quality synthetic views that respect the scene geometry, thus outperforming the state-of-the-art methods. Expand
Novel View Synthesis for Large-scale Scene using Adversarial Loss
This work proposes a novel fully convolutional network, that can take advantage of the structural information explicitly by incorporating the inverse depth features, and shows that this method can generate novel views of large-scale scene without distortion. Expand
Learning-based view synthesis for light field cameras
This paper proposes a novel learning-based approach to synthesize new views from a sparse set of input views that could potentially decrease the required angular resolution of consumer light field cameras, which allows their spatial resolution to increase. Expand
Deep Stereo: Learning to Predict New Views from the World's Imagery
This work presents a novel deep architecture that performs new view synthesis directly from pixels, trained from a large number of posed image sets, and is the first to apply deep learning to the problem ofnew view synthesis from sets of real-world, natural imagery. Expand
Deeper Depth Prediction with Fully Convolutional Residual Networks
A fully convolutional architecture, encompassing residual learning, to model the ambiguous mapping between monocular images and depth maps is proposed and a novel way to efficiently learn feature map up-sampling within the network is presented. Expand
Unsupervised Monocular Depth Estimation with Left-Right Consistency
This paper proposes a novel training objective that enables the convolutional neural network to learn to perform single image depth estimation, despite the absence of ground truth depth data, and produces state of the art results for monocular depth estimation on the KITTI driving dataset. Expand
Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis
A novel recurrent convolutional encoder-decoder network that is trained end-to-end on the task of rendering rotated objects starting from a single image and allows the model to capture long-term dependencies along a sequence of transformations. Expand
Depth synthesis and local warps for plausible image-based navigation
This work introduces a new IBR algorithm that is robust to missing or unreliable geometry, providing plausible novel views even in regions quite far from the input camera positions, and demonstrates novel view synthesis in real time for multiple challenging scenes with significant depth complexity. Expand
Self-Supervised Learning for Stereo Matching with Self-Improving Ability
A simple convolutional neural network architecture that is able to learn to compute dense disparity maps directly from the stereo inputs that outperforms many state-of-the-art stereo matching methods with a margin, and at the same time significantly faster. Expand