Monocular Reconstruction of Neural Face Reflectance Fields

@article{MallikarjunB2021MonocularRO,
  title={Monocular Reconstruction of Neural Face Reflectance Fields},
  author={R. MallikarjunB. and Ayush Tewari and Tae-Hyun Oh and Tim Weyrich and B. Bickel and Hans-Peter Seidel and Hanspeter Pfister and Wojciech Matusik and Mohamed A. Elgharib and Christian Theobalt},
  journal={2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2021},
  pages={4789-4798}
}
The reflectance field of a face describes the reflectance properties responsible for complex lighting effects including diffuse, specular, inter-reflection and self shadowing. Most existing methods for estimating the face reflectance from a monocular image assume faces to be diffuse with very few approaches adding a specular component. This still leaves out important perceptual aspects of reflectance such as higher-order global illumination effects and self-shadowing. We present a new neural… 

Figures and Tables from this paper

Learning Physically-based Material and Lighting Decompositions for Face Editing
TLDR
To allow fast and controllable reflectance and lighting editing, a physically-based decomposition through deep learned priors from path-traced portrait images is formed, which can better represent the true appearance function than simpler baseline methods, leading to better generalization and higher-quality editing.
AvatarMe++: Facial Shape and BRDF Inference with Photorealistic Rendering-Aware GANs
TLDR
This work introduces the first method that is able to reconstruct photorealistic render-ready 3D facial geometry and BRDF from a single '`in-the-wild'' image and reconstructs high-resolution 3D faces from asingle low-resolution image that can be rendered in various applications, and bridge the uncanny valley.
Deep Neural Models for Illumination Estimation and Relighting: A Survey
TLDR
This contribution aims to bring together in a coherent manner current advances in this conjunction, presented in three categories: scene illumination estimation, relighting with reflectance‐aware scene‐specific representations and finally relighting as image‐to‐image transformations.
Face Relighting with Geometrically Consistent Shadows
TLDR
This work proposes a novel differentiable algorithm for synthesizing hard shadows based on ray tracing, which it incorporates into training the face relighting model and demonstrates that this differentiable hard shadow modeling improves the quality of the estimated face geometry over diffuse shading models.
NeRF-Tex: Neural Reflectance Field Textures
TLDR
This work proposes to employ a versatile volumetric primitive represented by a neural reflectance field (NeRF-Tex), which jointly models the geometry of the material and its response to lighting, which unites the versatility and modeling power of neural networks with the artistic control needed for precise modeling of virtual scenes.
A Light Stage on Every Desk
TLDR
This paper proposes to use this time-varying illumination for synthetic relighting of your face with any new illumination condition, and trains a deep network on images plus monitor patterns of a given user and learn to predict images of that user under any target illumination (monitor pattern).
Neural Relighting and Expression Transfer On Video Portraits
TLDR
A neural relighting and expression transfer technique to transfer the head pose and facial expressions from a source performer to a portrait video of a target performer while enabling dynamic relighting.
Neural Video Portrait Relighting in Real-time via Consistency Modeling
TLDR
A neural approach for realtime, high-quality and coherent video portrait relighting, which jointly models the semantic, temporal and lighting consistency using a new dynamic OLAT dataset and adopts a temporal modeling scheme via flow-based supervision.
Relightable Neural Video Portrait
Figure 1. We introduce a relightable neural video portrait scheme for simultaneous relighting and reenactment that transfers the head pose and facial expressions from a source actor to a portrait
PhotoApp: Photorealistic Appearance Editing of Head Portraits
MALLIKARJUN B R and AYUSH TEWARI,MPI for Informatics, SIC, Germany ABDALLAH DIB, InterDigital R&I, France TIM WEYRICH, University College London, UK BERND BICKEL, IST Austria, Austria HANS-PETER
...
...

References

SHOWING 1-10 OF 48 REFERENCES
Acquiring the reflectance field of a human face
TLDR
A method to acquire the reflectance field of a human face and use these measurements to render the face under arbitrary changes in lighting and viewpoint and demonstrates the technique with synthetic renderings of a person's face under novel illumination and viewpoints.
High-fidelity facial reflectance and geometry inference from an unconstrained image
TLDR
A deep learning-based technique to infer high-quality facial reflectance and geometry given a single unconstrained image of the subject, which may contain partial occlusions and arbitrary illumination conditions, and demonstrates the rendering of high-fidelity 3D avatars from a variety of subjects captured under different lighting conditions.
Multiview face capture using polarized spherical gradient illumination
TLDR
A new pair of linearly polarized lighting patterns are presented which enables multiview diffuse-specular separation under a given spherical illumination condition from just two photographs, enabling more efficient acquisition of diffuse and specular albedo and normal maps from multiple viewpoints.
Self-Supervised Multi-level Face Model Learning for Monocular Reconstruction at Over 250 Hz
TLDR
This first approach that jointly learns a regressor for face shape, expression, reflectance and illumination on the basis of a concurrently learned parametric face model is presented, which compares favorably to the state-of-the-art in terms of reconstruction quality, better generalizes to real world faces, and runs at over 250 Hz.
A Morphable Face Albedo Model
TLDR
The model is demonstrated in a state of the art analysis-by-synthesis 3DMM fitting pipeline, are the first to integrate specular map estimation and outperform the Basel Face Model in albedo reconstruction.
Learning Detailed Face Reconstruction from a Single Image
TLDR
This work proposes to leverage the power of convolutional neural networks to produce a highly detailed face reconstruction from a single image, and introduces an end- to-end CNN framework which derives the shape in a coarse-to-fine fashion.
Analysis of human faces using a measurement-based skin reflectance model
We have measured 3D face geometry, skin reflectance, and subsurface scattering using custom-built devices for 149 subjects of varying age, gender, and race. We developed a novel skin reflectance
FML: Face Model Learning From Videos
TLDR
This work proposes multi-frame video-based self-supervised training of a deep network that learns a face identity model both in shape and appearance while jointly learning to reconstruct 3D faces.
Learning Physics-Guided Face Relighting Under Directional Light
TLDR
This work investigates end-to-end deep learning architectures that both de-light and relight an image of a human face, and decomposes the input image into intrinsic components according to a diffuse physics-based image formation model.
Reconstruction of Personalized 3D Face Rigs from Monocular Video
TLDR
A novel approach for the automatic creation of a personalized high-quality 3D face rig of an actor from just monocular video data, based on three distinct layers that allow the actor's facial shape as well as capture his person-specific expression characteristics at high fidelity, ranging from coarse-scale geometry to fine-scale static and transient detail on the scale of folds and wrinkles.
...
...