View Synthesis by Appearance Flow

@article{Zhou2016ViewSB,
  title={View Synthesis by Appearance Flow},
  author={Tinghui Zhou and Shubham Tulsiani and Weilun Sun and Jitendra Malik and Alexei A. Efros},
  journal={ArXiv},
  year={2016},
  volume={abs/1605.03557}
}
We address the problem of novel view synthesis: given an input image, synthesizing new images of the same object or scene observed from arbitrary viewpoints. We approach this as a learning task but, critically, instead of learning to synthesize pixels from scratch, we learn to copy them from the input image. Our approach exploits the observation that the visual appearance of different views of the same instance is highly correlated, and such correlation could be explicitly learned by training a… 

Figures and Tables from this paper

Generative Appearance Flow: A Hybrid Approach for Outdoor View Synthesis
TLDR
A novel convolutional neural network architecture is proposed that is trained without the need for additional geometric constraints, such as a ground-truth depth map, making it more broadly applicable.
Deep Learning based Novel View Synthesis
TLDR
A deep convolutional neural network (CNN) which learns to predict novel views of a scene from given collection of images and estimates, at each pixel, the probability distribution over possible depth levels in the scene.
Deep View Morphing
TLDR
A novel CNN architecture for view synthesis called Deep View Morphing that does not suffer from lack of texture details, shape distortions, or high computational complexity and significantly outperforms the state-of-the-art CNN-based view synthesis method.
Fast View Synthesis with Deep Stereo Vision
TLDR
A novel view synthesis approach based on stereo-vision and CNNs that decomposes the problem into two sub-tasks: view dependent geometry estimation and texture inpainting that could be effectively learned with CNNs is presented.
Novel View Synthesis for Large-scale Scene using Adversarial Loss
TLDR
This work proposes a novel fully convolutional network, that can take advantage of the structural information explicitly by incorporating the inverse depth features, and shows that this method can generate novel views of large-scale scene without distortion.
Multi-view to Novel View: Synthesizing Novel Views With Self-learned Confidence
TLDR
This paper proposes an end-to-end trainable framework that learns to exploit multiple viewpoints to synthesize a novel view without any 3D supervision, and introduces a self-learned confidence aggregation mechanism.
Novel View Synthesis With Skip Connections
  • Juhyeon Kim, Y. Kim
  • Computer Science
    2020 IEEE International Conference on Image Processing (ICIP)
  • 2020
TLDR
This paper investigates several variations of skip connection on two widely used novel view synthesis modules, pixel generation and flow prediction, and finds that the combination of the skip connections with flow-based hard attention is helpful.
Geometry-Aware Deep Network for Single-Image Novel View Synthesis
TLDR
A new region-aware geometric transform network is developed that performs these multiple tasks in a common framework and demonstrates the effectiveness of the network in generating high-quality synthetic views that respect the scene geometry, thus outperforming the state-of-the-art methods.
Novel View Synthesis via Depth-guided Skip Connections
TLDR
This paper utilizes an encoder–decoder architecture to regress pixels of a target view and couple the decoder aligned feature maps with skip connections, where the alignment is guided by predicted depth map of the target view.
View Depth Prediction Forward MappingReference Views Unseen View Camera Pose ( a ) Proxy geometry estimation and view warping ( b ) Texture
TLDR
A novel view synthesis approach based on stereo-vision and CNNs that decomposes the problem into two sub-tasks: view dependent geometry estimation and texture inpainting that could be effectively learned with CNNs is presented.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 42 REFERENCES
Single-view to Multi-view: Reconstructing Unseen Views with a Convolutional Network
TLDR
A convolutional network capable of generating images of a previously unseen object from arbitrary viewpoints given a single image of this object and an implicit 3D representation of the object class is presented.
Deep Stereo: Learning to Predict New Views from the World's Imagery
TLDR
This work presents a novel deep architecture that performs new view synthesis directly from pixels, trained from a large number of posed image sets, and is the first to apply deep learning to the problem ofnew view synthesis from sets of real-world, natural imagery.
3D-Assisted Image Feature Synthesis for Novel Views of an Object
TLDR
Experimental results show that the synthesized features of this paper enable view-independent comparison between images and perform significantly better than traditional image features in this respect.
3D-Assisted Feature Synthesis for Novel Views of an Object
TLDR
Experimental results show that the synthesized features of this paper enable view-independent comparison between images and perform significantly better than other traditional approaches in this respect.
Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis
TLDR
A novel recurrent convolutional encoder-decoder network that is trained end-to-end on the task of rendering rotated objects starting from a single image and allows the model to capture long-term dependencies along a sequence of transformations.
Deep Convolutional Inverse Graphics Network
This paper presents the Deep Convolution Inverse Graphics Network (DC-IGN), a model that aims to learn an interpretable representation of images, disentangled with respect to three-dimensional scene
Learning Image Representations Tied to Egomotion from Unlabeled Video
TLDR
This work proposes a new “embodied” visual learning paradigm, exploiting proprioceptive motor signals to train visual representations from egocentric video with no manual supervision, and shows that this unsupervised feature learning approach significantly outperforms previous approaches on visual recognition and next-best-view prediction tasks.
Multi-view 3D Models from Single Images with a Convolutional Network
TLDR
A convolutional network capable of inferring a 3D representation of a previously unseen object given a single image of this object and several depth maps fused together give a full point cloud of the object.
Learning Image Representations Tied to Ego-Motion
TLDR
This work proposes to exploit proprioceptive motor signals to provide unsupervised regularization in convolutional neural networks to learn visual representations from egocentric video to enforce that the authors' learned features exhibit equivariance, i.e, they respond predictably to transformations associated with distinct ego-motions.
Novel Views of Objects from a Single Image
TLDR
This work shows how to improve novel-view synthesis by making use of the correlations observed in 3D models and applying them to new image instances, and proposes a technique to use the structural information extracted from a 3D model that matches the image object in terms of viewpoint and shape.
...
1
2
3
4
5
...