Robust Pose Transfer With Dynamic Details Using Neural Video Rendering

@article{Sun2021RobustPT,
  title={Robust Pose Transfer With Dynamic Details Using Neural Video Rendering},
  author={Yang-tian Sun and Haozhi Huang and Xuan Wang and Yu-Kun Lai and Wei Liu and Lin Gao},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year={2021},
  volume={45},
  pages={2660-2666}
}
Pose transfer of human videos aims to generate a high-fidelity video of a target person imitating actions of a source person. A few studies have made great progress either through image translation with deep latent features or neural rendering with explicit 3D features. However, both of them rely on large amounts of training data to generate realistic results, and the performance degrades on more accessible Internet videos due to insufficient training frames. In this paper, we demonstrate that… 

Deep Person Generation: A Survey from the Perspective of Face, Pose and Cloth Synthesis

The scope of person generation is summarized, and recent progress and technical trends in identity-preserving deep person generation are reviewed, covering three major tasks: talking-head generation (face), pose-guided person generation (pose) and garment-oriented persongeneration (cloth).

References

SHOWING 1-10 OF 37 REFERENCES

Neural Human Video Rendering by Learning Dynamic Textures and Rendering-to-Video Translation.

A novel human video synthesis method that approaches limiting factors by explicitly disentangling the learning of time-coherent fine-scale details from the embedding of the human in 2D screen space and shows significant improvement over the state of the art both qualitatively and quantitatively.

Neural Rendering and Reenactment of Human Actor Videos

The proposed method for generating video-realistic animations of real humans under user control relies on a video sequence in conjunction with a (medium-quality) controllable 3D template model of the person to generate a synthetically rendered version of the video.

Real-Time Neural Style Transfer for Videos

  • Haozhi HuangHao Wang W. Liu
  • Computer Science
    2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2017
This work proposes a hybrid loss to capitalize on the content information of input frames, the style information of a given style image, and the temporal information of consecutive frames to calculate the temporal loss during the training stage.

High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs

A new method for synthesizing high-resolution photo-realistic images from semantic label maps using conditional generative adversarial networks (conditional GANs) is presented, which significantly outperforms existing methods, advancing both the quality and the resolution of deep image synthesis and editing.

Human Motion Transfer with 3D Constraints and Detail Enhancement

We propose a new method for realistic human motion transfer using a generative adversarial network (GAN), which generates a motion video of a target character imitating actions of a source character,

Textured Neural Avatars

A system for learning full body neural avatars, i.e. deep networks that produce full body renderings of a person for varying body pose and varying camera pose, that is capable of learning to generate realistic renderings while being trained on videos annotated with 3D poses and foreground masks is presented.

Deferred Neural Rendering: Image Synthesis using Neural Textures

This work proposes Neural Textures, which are learned feature maps that are trained as part of the scene capture process that can be utilized to coherently re-render or manipulate existing video content in both static and dynamic environments at real-time rates.

Deferred neural rendering

This work proposes Neural Textures, which are learned feature maps that are trained as part of the scene capture process that can be utilized to coherently re-render or manipulate existing video content in both static and dynamic environments at real-time rates.

Animating Arbitrary Objects via Deep Motion Transfer

This paper introduces a novel deep learning framework for image animation that generates a video in which the target object is animated according to the driving sequence through a deep architecture that decouples appearance and motion information.

Dense Pose Transfer

This work proposes a combination of surface-based pose estimation and deep generative models that allows us to perform accurate pose transfer, i.e. synthesize a new image of a person based on a single image of that person and theimage of a pose donor.