• Corpus ID: 245537311

Human View Synthesis using a Single Sparse RGB-D Input

@article{Nguyen2021HumanVS,
  title={Human View Synthesis using a Single Sparse RGB-D Input},
  author={Phong Nguyen and Nikolaos Sarafianos and Christoph Lassner and J. Heikkila and Tony Tung},
  journal={ArXiv},
  year={2021},
  volume={abs/2112.13889}
}
Novel view synthesis for humans in motion is a challenging computer vision problem that enables applications such as free-viewpoint video. Existing methods typically use complex setups with multiple input views, 3D supervision or pre-trained models that do not generalize well to new identities. Aiming to address these limitations, we present a novel view synthesis framework to generate realistic renders from unseen views of any human captured from a single-view sensor with sparse RGB-D, similar… 

Figures and Tables from this paper

Learning Dynamic View Synthesis With Few RGBD Cameras
TLDR
This work proposes to utilize RGBD cameras to remove limitations and synthesize free-viewpoint videos of dynamic indoor scenes and introduces a simple Regional Depth-Inpainting module that adaptively inpaints missing depth values to render complete novel views.
Animatable Neural Radiance Fields from Monocular RGB-D
TLDR
This paper introduces a novel method to integrate observations across frames and encode the appearance at each individual frame by utilizing the human pose that models the body shape and point clouds which cover partial part of the human as the input.

References

SHOWING 1-10 OF 78 REFERENCES
Deep Volumetric Video From Very Sparse Multi-view Performance Capture
TLDR
This work focuses on the task of template-free, per-frame 3D surface reconstruction from as few as three RGB sensors, for which conventional visual hull or multi-view stereo methods fail to generate plausible results.
Free View Synthesis
TLDR
This work presents a method for novel view synthesis from input images that are freely distributed around a scene that can synthesize images for free camera movement through the scene, and works for general scenes with unconstrained geometric layouts.
View Synthesis by Appearance Flow
TLDR
This work addresses the problem of novel view synthesis: given an input image, synthesizing new images of the same object or scene observed from arbitrary viewpoints and shows that for both objects and scenes, this approach is able to synthesize novel views of higher perceptual quality than previous CNN-based techniques.
Stereo Magnification: Learning View Synthesis using Multiplane Images
TLDR
This paper explores an intriguing scenario for view synthesis: extrapolating views from imagery captured by narrow-baseline stereo cameras, including VR cameras and now-widespread dual-lens camera phones, and proposes a learning framework that leverages a new layered representation that is called multiplane images (MPIs).
LookinGood: Enhancing Performance Capture with Real-time Neural Re-Rendering
TLDR
The novel approach to augment such real-time performance capture systems with a deep architecture that takes a rendering from an arbitrary viewpoint, and jointly performs completion, super resolution, and denoising of the imagery in real- time is taken.
Learning-based view synthesis for light field cameras
TLDR
This paper proposes a novel learning-based approach to synthesize new views from a sparse set of input views that could potentially decrease the required angular resolution of consumer light field cameras, which allows their spatial resolution to increase.
Depth synthesis and local warps for plausible image-based navigation
TLDR
This work introduces a new IBR algorithm that is robust to missing or unreliable geometry, providing plausible novel views even in regions quite far from the input camera positions, and demonstrates novel view synthesis in real time for multiple challenging scenes with significant depth complexity.
IGNOR: Image-guided Neural Object Rendering
TLDR
A learned image-guided rendering technique that combines the benefits of image-based rendering and GAN-based image synthesis to generate photo-realistic re-renderings of reconstructed objects for virtual and augmented reality applications.
Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenes
TLDR
Stereo Radiance Fields is introduced, a neural view synthesis approach that is trained end-to-end, generalizes to new scenes, and requires only sparse views at test time, andExperiments show that SRF learns structure instead of over-fitting on a scene, achieving significantly sharper, more detailed results than scene-specific models.
Novel View Synthesis from Single Images via Point Cloud Transformation
TLDR
The argument is made that for true novel view synthesis of objects, where the object can be synthesized from any viewpoint, an explicit 3D shape representation is needed, and point clouds are estimated to capture the geometry of the object.
...
1
2
3
4
5
...