Corpus ID: 235458378

To fit or not to fit: Model-based Face Reconstruction and Occlusion Segmentation from Weak Supervision

  title={To fit or not to fit: Model-based Face Reconstruction and Occlusion Segmentation from Weak Supervision},
  author={Chunlu Li and Andreas Morel-Forster and T. Vetter and B. Egger and Adam Kortylewski},
3D face reconstruction from a single image is challenging due to its ill-posed nature. Model-based face autoencoders address this issue effectively by fitting a face model to the target image in a weakly supervised manner. However, in unconstrained environments occlusions distort the face reconstruction because the model often erroneously tries to adapt to occluded face regions. Supervised occlusion segmentation is a viable solution to avoid the fitting of occluded face regions, but it requires… Expand


Occlusion-Aware 3D Morphable Models and an Illumination Prior for Face Image Analysis
This work proposes a fully automated, probabilistic and occlusion-aware 3D morphable face model adaptation framework following an analysis-by-synthesis setup and proposes a RANSAC-based robust illumination estimation technique. Expand
Self-Supervised Multi-level Face Model Learning for Monocular Reconstruction at Over 250 Hz
This first approach that jointly learns a regressor for face shape, expression, reflectance and illumination on the basis of a concurrently learned parametric face model is presented, which compares favorably to the state-of-the-art in terms of reconstruction quality, better generalizes to real world faces, and runs at over 250 Hz. Expand
Accurate 3D Face Reconstruction With Weakly-Supervised Learning: From Single Image to Image Set
A novel deep 3D face reconstruction approach that leverages a robust, hybrid loss function for weakly-supervised learning which takes into account both low-level and perception-level information for supervision, and performs multi-image face reconstruction by exploiting complementary information from different images for shape aggregation is proposed. Expand
Learning to Regress 3D Face Shape and Expression From an Image Without 3D Supervision
To train a network without any 2D-to-3D supervision, RingNet is presented, which learns to compute 3D face shape from a single image and achieves invariance to expression by representing the face using the FLAME model. Expand
"Look Ma, No Landmarks!" - Unsupervised, Model-Based Dense Face Alignment
This paper shows how to train an image-to-image network to predict dense correspondence between a face image and a 3D morphable model using only the model for supervision and shows that both geometric parameters and photometric parameters can be inferred directly from the correspondence map using linear least squares and the novel inverse spherical harmonic lighting model. Expand
Learning Detailed Face Reconstruction from a Single Image
This work proposes to leverage the power of convolutional neural networks to produce a highly detailed face reconstruction from a single image, and introduces an end- to-end CNN framework which derives the shape in a coarse-to-fine fashion. Expand
MoFA: Model-Based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction
A novel model-based deep convolutional autoencoder that addresses the highly challenging problem of reconstructing a 3D human face from a single in-the-wild color image and can be trained end-to-end in an unsupervised manner, which renders training on very large real world data feasible. Expand
Real-Time Facial Segmentation and Performance Capture from RGB Input
A state-of-the-art regression-based facial tracking framework with segmented face images as training is adopted, and accurate and uninterrupted facial performance capture is demonstrated in the presence of extreme occlusion and even side views. Expand
Learning an animatable detailed 3D face model from in-the-wild images
This work presents the first approach that regresses 3D face shape and animatable details that are specific to an individual but change with expression, and introduces a novel detail-consistency loss that disentangles person-specific details from expression-dependent wrinkles. Expand
Towards High-Fidelity Nonlinear 3D Face Morphable Model
A novel approach to learn additional proxies as means to side-step strong regularizations, as well as, leverages to promote detailed shape/albedo on 3D face reconstruction by solely optimizing latent representations is presented. Expand