AvatarMe: Realistically Renderable 3D Facial Reconstruction “In-the-Wild”

@article{Lattas2020AvatarMeRR,
  title={AvatarMe: Realistically Renderable 3D Facial Reconstruction “In-the-Wild”},
  author={Alexandros Lattas and Stylianos Moschoglou and Baris Gecer and Stylianos Ploumpis and Vasileios Triantafyllou and Abhijeet Ghosh and Stefanos Zafeiriou},
  journal={2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2020},
  pages={757-766}
}
Over the last years, with the advent of Generative Adversarial Networks (GANs), many face analysis tasks have accomplished astounding performance, with applications including, but not limited to, face generation and 3D face reconstruction from a single "in-the-wild" image. Nevertheless, to the best of our knowledge, there is no method which can produce high-resolution photorealistic 3D faces from "in-the-wild" images and this can be attributed to the: (a) scarcity of available data for training… 

Figures and Tables from this paper

AvatarMe++: Facial Shape and BRDF Inference with Photorealistic Rendering-Aware GANs

TLDR
This work introduces the first method that is able to reconstruct photorealistic render-ready 3D facial geometry and BRDF from a single '`in-the-wild'' image and reconstructs high-resolution 3D faces from asingle low-resolution image that can be rendered in various applications, and bridge the uncanny valley.

3D human tongue reconstruction from single "in-the-wild" images

TLDR
This work presents the first, to the best of the knowledge, end-to-end trainable pipeline that accurately reconstructs the 3D face together with the tongue and makes this pipeline robust in “in-the-wild” images by introducing a novel GAN method tailored for 3D tongue surface generation.

Inverting Generative Adversarial Renderer for Face Reconstruction

TLDR
This work introduces a novel Generative Adversarial Renderer (GAR) and proposes to tailor its inverted version to the general fitting pipeline, to tackle the above problem of domain-shift noise in training and optimization.

Fast-GANFIT: Generative Adversarial Network for High Fidelity 3D Face Reconstruction

TLDR
This paper utilizes GANs to train a very powerful facial texture prior from a large-scale 3D texture dataset and proposes a novel self-supervised regression based approach that achieves for the first time, to the best of the authors' knowledge, facial texture reconstruction with high-frequency details.

Synthesizing Coupled 3D Face Modalities by Trunk-Branch Generative Adversarial Networks

TLDR
This paper presents the first methodology that generates high-quality texture, shape, and normals jointly jointly, which can be used for photo-realistic synthesis and proposes a novel GAN that can generate data from different modalities while exploiting their correlations.

OSTeC: One-Shot Texture Completion

TLDR
This work proposes an unsupervised approach for one-shot 3D facial texture completion that does not re-quire large-scale texture datasets, but rather harnesses the knowledge stored in 2D face generators to improve pose-invariant face recognition.

3DMM-RF: Convolutional Radiance Fields for 3D Face Modeling

TLDR
This work presents a facial 3D Morphable Model, which can accurately model a subject’s identity, pose and expression and render it in arbitrary illumination, and introduces a style-based generative network that synthesizes in one pass all and only the required rendering samples of a neural radiance.

Controllable 3D Generative Adversarial Face Model via Disentangling Shape and Appearance

TLDR
A new 3D face generative model that can decouple identity and expression and provides granular control over expressions is proposed, using a pair of supervised auto-encoder and generative adversarial networks to produce high-quality 3D faces, both in terms of appearance and shape.

Self-supervised Re-renderable Facial Albedo Reconstruction from Single Image

TLDR
This paper proposes a new self-supervised deep learning framework for reconstructing high-quality and re- renderable facial albedos from single-view images in-the-wild and designs several novel regularization losses on both the albedo and illumination maps to facilitate the disentanglement of these two factors.

StyleUV: Diverse and High-fidelity UV Map Generative Model

TLDR
A novel UV map generative model that learns to generate diverse and realistic synthetic UV maps without requiring high-quality UV maps for training is presented.
...

References

SHOWING 1-10 OF 46 REFERENCES

GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction

TLDR
This paper utilizes GANs to train a very powerful generator of facial texture in UV space and revisits the original 3D Morphable Models (3DMMs) fitting approaches making use of non-linear optimization to find the optimal latent parameters that best reconstruct the test image but under a new perspective.

Synthesizing Coupled 3D Face Modalities by Trunk-Branch Generative Adversarial Networks

TLDR
This paper presents the first methodology that generates high-quality texture, shape, and normals jointly jointly, which can be used for photo-realistic synthesis and proposes a novel GAN that can generate data from different modalities while exploiting their correlations.

Synthesizing Coupled 3 D Face Modalities by Trunk-Branch Generative Adversarial Networks

TLDR
This paper presents the first methodology that generates high-quality texture, shape, and normals jointly jointly, which can be used for photo-realistic synthesis and proposes a novel GAN that can generate data from different modalities while exploiting their correlations.

3D Face Reconstruction by Learning from Synthetic Data

TLDR
The proposed approach is based on a Convolutional-Neural-Network (CNN) which extracts the face geometry directly from its image and successfully recovers facial shapes from real images, even for faces with extreme expressions and under various lighting conditions.

3D Face Morphable Models "In-the-Wild"

TLDR
This paper proposes the first, to the best of the knowledge, in-the-wild 3DMM by combining a powerful statistical model of facial shape, which describes both identity and expression, with an in- the-wild texture model, and demonstrates the first 3D facial database with relatively unconstrained conditions.

Neural Face Editing with Intrinsic Image Disentangling

TLDR
An end-to-end generative adversarial network is proposed that infers a face-specific disentangled representation of intrinsic face properties, including shape, albedo, and lighting, and an alpha matte, and it is shown that this network can be trained on in thewild images by incorporating an in-network physically-based image formation module and appropriate loss functions.

Regressing Robust and Discriminative 3D Morphable Models with a Very Deep Neural Network

TLDR
A robust method for regressing discriminative 3D morphable face models (3DMM) using a convolutional neural network to regress 3DMM shape and texture parameters directly from an input photo is described.

Photorealistic Facial Texture Inference Using Deep Neural Networks

TLDR
A data-driven inference method is presented that can synthesize a photorealistic texture map of a complete 3D face model given a partial 2D view of a person in the wild and successful face reconstructions from a wide range of low resolution input images are demonstrated.

Self-Supervised Multi-level Face Model Learning for Monocular Reconstruction at Over 250 Hz

TLDR
This first approach that jointly learns a regressor for face shape, expression, reflectance and illumination on the basis of a concurrently learned parametric face model is presented, which compares favorably to the state-of-the-art in terms of reconstruction quality, better generalizes to real world faces, and runs at over 250 Hz.

On Learning 3D Face Morphable Model from In-the-Wild Images

  • Luan TranXiaoming Liu
  • Computer Science
    IEEE Transactions on Pattern Analysis and Machine Intelligence
  • 2021
TLDR
This paper proposes an innovative framework to learn a nonlinear 3DMM model from a large set of in-the-wild face images, without collecting 3D face scans, and demonstrates the superior representation power of the nonlinear3DMM over its linear counterpart, and its contribution to face alignment, 3D reconstruction, and face editing.