Inverting Generative Adversarial Renderer for Face Reconstruction

@article{Piao2021InvertingGA,
  title={Inverting Generative Adversarial Renderer for Face Reconstruction},
  author={Jingtan Piao and Keqiang Sun and Kwan-Yee Lin and Hongshneg Li},
  journal={2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2021},
  pages={15614-15623}
}
Given a monocular face image as input, 3D face geometry reconstruction aims to recover a corresponding 3D face mesh. Recently, both optimization-based and learning-based face reconstruction methods have taken advantage of the emerging differentiable renderer and shown promising results. However, the differentiable renderer, mainly based on graphics rules, simplifies the realistic mechanism of the illumination, reflection, etc., of the real world, thus can-not produce realistic images. This… 

Figures and Tables from this paper

REALY: Rethinking the Evaluation of 3D Face Reconstruction
TLDR
A novel evaluation approach with a new benchmark REALY, consists of 100 globally aligned face scans with accurate facial keypoints, high-quality region masks, and topologyconsistent meshes that performs region-wise shape alignment and leads to more accurate, bidirectional correspondences during computing the shape errors.
Controllable 3D Face Synthesis with Conditional Generative Occupancy Fields
TLDR
A new NeRF-based conditional 3D face synthesis framework is proposed, which enables 3D controllability over the generated face images by imposing explicit 3D conditions from3D face priors and effectively enforces the shape of thegenerated face to commit to a given 3D Morphable Model (3DMM) mesh.
MOST-GAN: 3D Morphable StyleGAN for Disentangled Face Image Manipulation
TLDR
MOST-GAN achieves photorealistic manipulation of portrait images with fully disentangled 3D control over their physical attributes, enabling extreme manipulation of lighting, facial expression, and pose variations up to full profile view.

References

SHOWING 1-10 OF 53 REFERENCES
GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction
TLDR
This paper utilizes GANs to train a very powerful generator of facial texture in UV space and revisits the original 3D Morphable Models (3DMMs) fitting approaches making use of non-linear optimization to find the optimal latent parameters that best reconstruct the test image but under a new perspective.
A Style-Based Generator Architecture for Generative Adversarial Networks
TLDR
An alternative generator architecture for generative adversarial networks is proposed, borrowing from style transfer literature, that improves the state-of-the-art in terms of traditional distribution quality metrics, leads to demonstrably better interpolation properties, and also better disentangles the latent factors of variation.
SfSNet: Learning Shape, Reflectance and Illuminance of Faces 'in the Wild'
TLDR
SfSNet produces significantly better quantitative and qualitative results than state-of-the-art methods for inverse rendering and independent normal and illumination estimation and is designed to reflect a physical lambertian rendering model.
Progressive Growing of GANs for Improved Quality, Stability, and Variation
TLDR
A new training methodology for generative adversarial networks is described, starting from a low resolution, and adding new layers that model increasingly fine details as training progresses, allowing for images of unprecedented quality.
AOT: Appearance Optimal Transport Based Identity Swapping for Forgery Detection
TLDR
This work provides a new identity swapping algorithm with large differences in appearance for face forgery detection and proposes an Appearance Optimal Transport model (AOT) to formulate it in both latent and pixel space.
MEAD: A Large-Scale Audio-Visual Dataset for Emotional Talking-Face Generation
TLDR
The Multi-view Emotional Audio-visual Dataset (MEAD) is built, a talking-face video corpus featuring 60 actors and actresses talking with eight different emotions at three different intensity levels that could benefit a number of different research fields including conditional generation, cross-modal understanding and expression recognition.
ReDA:Reinforced Differentiable Attribute for 3D Face Reconstruction
TLDR
This work introduces a new free-form deformation layer that sits on top of 3DMM to enjoy both the prior knowledge and out-of-space modeling, and makes the rendering to be more differentiable through a set of convolution operations with multi-scale kernel sizes.
Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning
TLDR
An approach for face image generation of virtual people with disentangled, precisely-controllable latent representations for identity of non-existing people, expression, pose, and illumination and introduces contrastive learning to promote disentanglement.
AvatarMe: Realistically Renderable 3D Facial Reconstruction “In-the-Wild”
TLDR
The first method that is able to reconstruct photorealistic 3D faces from a single "in-the-wild" image with an increasing level of detail is introduced, which outperforms the existing arts by a significant margin and reconstructs authentic, 4K by 6K-resolution 3D Faces from asingle low-resolution image that, for the first time, bridges the uncanny valley.
Analyzing and Improving the Image Quality of StyleGAN
TLDR
This work redesigns the generator normalization, revisit progressive growing, and regularize the generator to encourage good conditioning in the mapping from latent codes to images, and thereby redefines the state of the art in unconditional image modeling.
...
...