CNN-Based Real-Time Dense Face Reconstruction with Inverse-Rendered Photo-Realistic Face Images

@article{Guo2019CNNBasedRD,
  title={CNN-Based Real-Time Dense Face Reconstruction with Inverse-Rendered Photo-Realistic Face Images},
  author={Yudong Guo and Juyong Zhang and Jianfei Cai and Boyi Jiang and Jianmin Zheng},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year={2019},
  volume={41},
  pages={1294-1307}
}
With the powerfulness of convolution neural networks (CNN), CNN based face reconstruction has recently shown promising performance in reconstructing detailed face shape from 2D face images. The success of CNN-based methods relies on a large number of labeled data. The state-of-the-art synthesizes such data using a coarse morphable face model, which however has difficulty to generate detailed photo-realistic images of faces (with wrinkles). This paper presents a novel face data generation method… 
View on IEEE
arxiv.org
102 Citations
Highly Influential Citations
9
Background Citations
34
Methods Citations
48
Results Citations
2

102 Citations

Citation Type

Citation Type

Learning-detailed 3D face reconstruction based on convolutional neural networks from a single image
TLDR
By using a learning-based inverse face rendering, this article proposes a novel data-generation technique by rendering a large number of face images that are photo-realistic and possess distinct properties and demonstrates the efficiency of this technique to pose, expression and lighting dynamics.
Lightweight Photometric Stereo for Facial Details Recovery
TLDR
This paper proposes a novel neural network specially designed for photometric stereo based 3D face reconstruction that only requires sparse inputs or even a single image to recover high-fidelity face shapes with images captured under near-field lights.
Accurate 3D Face Reconstruction With Weakly-Supervised Learning: From Single Image to Image Set
TLDR
A novel deep 3D face reconstruction approach that leverages a robust, hybrid loss function for weakly-supervised learning which takes into account both low-level and perception-level information for supervision, and performs multi-image face reconstruction by exploiting complementary information from different images for shape aggregation is proposed.
Facial Geometric Detail Recovery via Implicit Representation
TLDR
This work presents a robust texture-guided geometric detail recovery approach using only a single in-the-wild facial image and registers the implicit shape details to a 3D Morphable Model template, which can be used in traditional modeling and rendering pipelines.
Occlusion Resistant Network for 3D Face Reconstruction
TLDR
A novel context-learning-based distillation approach to tackle the occlusions in the face images, which uses a weak model (unsuitable for occluded face images) to train a highly robust network towards partially and fully-occluded face images.
3D Face From X: Learning Face Shape From Diverse Sources
TLDR
Experimental results demonstrate that with training data from more sources, the proposed method to learn a unified face model can learn a more powerful face model.
Real-time hierarchical facial performance capture
TLDR
This paper presents a novel method to reconstruct high resolution facial geometry and appearance in real-time by capturing an individual-specific face model with fine-scale details, based on monocular RGB video input, and builds a hierarchical face pyramid through adaptive subdivisions and progressive refinements of the mesh from a coarse level to a fine level.
3D Single Image Face Reconstruction Approaches With Deep Neural Networks
TLDR
3D facial reconstruction algorithms used for application such as 3D printing, 3D VR games and facial recognition, and 3D face learning methods developed in 2019 are discussed.
Real-time Face Video Swapping From A Single Portrait
TLDR
A novel high-fidelity real-time method to replace the face in a target video clip by the face from a single source portrait image, unlike existing deep learning based methods, which does not need to pre-train any models.
Photo-Realistic Facial Details Synthesis From Single Image
We present a single-image 3D face synthesis technique that can handle challenging facial expressions while recovering fine geometric details. Our technique employs expression analysis for proxy face…
...
...

References

SHOWING 1-10 OF 66 REFERENCES
3D Face Reconstruction by Learning from Synthetic Data
TLDR
The proposed approach is based on a Convolutional-Neural-Network (CNN) which extracts the face geometry directly from its image and successfully recovers facial shapes from real images, even for faces with extreme expressions and under various lighting conditions.
3D Face Reconstruction With Geometry Details From a Single Image
TLDR
This work develops a novel method for reconstructing 3D faces from unconstrained 2D images using a coarse-to-fine optimization strategy that outperforms the state-of-the-art approaches in terms of accuracy and detail recovery.
InverseFaceNet: Deep Single-Shot Inverse Face Rendering From A Single Image
TLDR
This work proposes to recover high-quality facial pose, shape, expression, reflectance and illumination using a deep neural network that is trained using a large, synthetically created dataset and builds on a novel loss function that measures model-space similarity directly in parameter space and significantly improves reconstruction accuracy.
Learning Detailed Face Reconstruction from a Single Image
TLDR
This work proposes to leverage the power of convolutional neural networks to produce a highly detailed face reconstruction from a single image, and introduces an end- to-end CNN framework which derives the shape in a coarse-to-fine fashion.
Large Pose 3D Face Reconstruction from a Single Image via Direct Volumetric CNN Regression
TLDR
This work proposes to address many of these limitations by training a Convolutional Neural Network (CNN) on an appropriate dataset consisting of 2D images and 3D facial models or scans, and achieves this via a simple CNN architecture that performs direct regression of a volumetric representation of the3D facial geometry from a single 2D image.
InverseFaceNet: Deep Monocular Inverse Face Rendering
TLDR
This work introduces InverseFaceNet, a deep convolutional inverse rendering framework for faces that jointly estimates facial pose, shape, expression, reflectance and illumination from a single input image and demonstrates that this strategy outperforms completely synthetically trained networks.
Reconstruction of Personalized 3D Face Rigs from Monocular Video
TLDR
A novel approach for the automatic creation of a personalized high-quality 3D face rig of an actor from just monocular video data, based on three distinct layers that allow the actor's facial shape as well as capture his person-specific expression characteristics at high fidelity, ranging from coarse-scale geometry to fine-scale static and transient detail on the scale of folds and wrinkles.
MoFA: Model-Based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction
TLDR
A novel model-based deep convolutional autoencoder that addresses the highly challenging problem of reconstructing a 3D human face from a single in-the-wild color image and can be trained end-to-end in an unsupervised manner, which renders training on very large real world data feasible.
Reconstructing detailed dynamic face geometry from monocular video
TLDR
This work presents a new method for capturing face geometry from monocular video that captures detailed, dynamic, spatio-temporally coherent 3D face geometry without the need for markers, and successfully reconstructs expressive motion including high-frequency face detail such as folds and laugh lines.
Large-Pose Face Alignment via CNN-Based Dense 3D Model Fitting
TLDR
This paper proposes a face alignment method for large-pose face images, by combining the powerful cascaded CNN regressor method and 3DMM, and forms the face alignment as a3DMM fitting problem, where the camera projection matrix and3D shape parameters are estimated by a cascade of CNN-based regressors.
...
...