Faces as Lighting Probes via Unsupervised Deep Highlight Extraction

@inproceedings{Yi2018FacesAL,
  title={Faces as Lighting Probes via Unsupervised Deep Highlight Extraction},
  author={Renjiao Yi and Chenyang Zhu and Ping Tan and Stephen Lin},
  booktitle={ECCV},
  year={2018}
}
We present a method for estimating detailed scene illumination using human faces in a single image. [] Key Method Based on the observation that faces can exhibit strong highlight reflections from a broad range of lighting directions, we propose a deep neural network for extracting highlights from faces, and then trace these reflections back to the scene to acquire the environment map.
View PDF on arXiv
27 Citations
Highly Influential Citations
2
Background Citations
8
Methods Citations
8
Results Citations
1

27 Citations

Citation Type

Citation Type

Hybrid Face Reflectance, Illumination, and Shape from a Single Image.
TLDR
This work proposes HyFRIS-Net to jointly estimate the hybrid reflectance and illumination models, as well as the refined face shape from a single unconstrained face image in a pre-defined texture space to ensure photometric face appearance modeling in both parametric and non-parametric spaces for efficient learning.
Highlight Removal in Facial Images
TLDR
This work adopts the structure of conditional generative adversarial network (CGAN) to generate highlight-free images in facial images, which is, to the best knowledge, the largest image dataset for facial highlight removal.
Spec-Net and Spec-CGAN: Deep learning models for specularity removal from faces
High-Dynamic-Range Lighting Estimation From Face Portraits
TLDR
It is shown that the predicted HDR environment maps can be used as accurate illumination sources for scene renderings, with potential applications in 3D object insertion for augmented reality.
Leveraging Multi-view Image Sets for Unsupervised Intrinsic Image Decomposition and Highlight Separation
TLDR
An unsupervised approach for factorizing object appearance into highlight, shading, and albedo layers, trained by multi-view real images is presented, with a proposed image representation based on local color distributions that allows training to be insensitive to the local misalignments of multi-View images.
Deep Neural Models for Illumination Estimation and Relighting: A Survey
TLDR
This contribution aims to bring together in a coherent manner current advances in this conjunction, presented in three categories: scene illumination estimation, relighting with reflectance‐aware scene‐specific representations and finally relighting as image‐to‐image transformations.
Object-based Illumination Estimation with Rendering-aware Neural Networks
TLDR
An approach that takes advantage of physical principles from inverse rendering to constrain the solution, while also utilizing neural networks to expedite the more computationally expensive portions of its processing, to increase robustness to noisy input data as well as to improve temporal and spatial stability is proposed.
Learning Illumination from Diverse Portraits
TLDR
This work presents a learning-based technique for estimating high dynamic range, omnidirectional illumination from a single low dynamic range portrait image captured under arbitrary indoor or outdoor lighting conditions, and shows that this technique outperforms the state-of-the-art technique for portrait-based lighting estimation.
Learning Scene Illumination by Pairwise Photos from Rear and Front Mobile Cameras
TLDR
A learning based method to recover low‐frequency scene illumination represented as spherical harmonic functions by pairwise photos from rear and front cameras on mobile devices is proposed and produces visually and quantitatively superior results compared to the state‐of‐the‐arts.
Outdoor illumination estimation via all convolutional neural networks
...
1
2
3
...

References

SHOWING 1-10 OF 55 REFERENCES
Real-time illumination estimation from faces for coherent rendering
  • S. Knorr, Daniel Kurz
  • Computer Science
    2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)
  • 2014
TLDR
The approach provides plausible results, which considerably enhance the visual realism in real-time Augmented Reality applications, and recovers the most plausible real-world lighting conditions for measured reflections in a face, represented by a function depending on incident light angle using Spherical Harmonics.
From Faces to Outdoor Light Probes
TLDR
This paper presents an approach to directly estimate an HDR light probe from a single LDR photograph, shot outdoors with a consumer camera, without specialized calibration targets or equipment, and shows that relighting objects with HDR light probes estimated by the method yields realistic results in a wide variety of settings.
Deep Reflectance Maps
TLDR
A convolutional neural architecture to estimate reflectance maps of specular materials in natural lighting conditions is proposed in an end-to-end learning formulation that directly predicts a reflectance map from the image itself.
Learning to predict indoor illumination from a single image
TLDR
An end-to-end deep neural network is trained that directly regresses a limited field-of-view photo to HDR illumination, without strong assumptions on scene geometry, material properties, or lighting, which allows to automatically recover high-quality HDR illumination estimates that significantly outperform previous state- of-the-art methods.
Deep Outdoor Illumination Estimation
TLDR
It is demonstrated that the approach allows the recovery of plausible illumination conditions and enables photorealistic virtual object insertion from a single image and significantly outperforms previous solutions to this problem.
Specular Highlight Removal in Facial Images
TLDR
This work employs a melanin and hemoglobin based model to represent the diffuse color variations in facial skin, and utilizes this model to constrain the highlight removal solution in a manner that is effective even for partially saturated pixels.
Illumination estimation and cast shadow detection through a higher-order graphical model
TLDR
A higher-order Markov Random Field illumination model, which combines low-level shadow evidence with high-level prior knowledge for the joint estimation of cast shadows and the illumination environment, is described.
Learning Detailed Face Reconstruction from a Single Image
TLDR
This work proposes to leverage the power of convolutional neural networks to produce a highly detailed face reconstruction from a single image, and introduces an end- to-end CNN framework which derives the shape in a coarse-to-fine fashion.
Illumination from Shadows
TLDR
The effectiveness of using occluding information of incoming light in estimating an illumination distribution of a scene is demonstrated and an adaptive sampling framework for efficient estimation of illumination distribution is introduced.
Reflectance and Illumination Recovery in the Wild
TLDR
A reflectance model and priors are developed that precisely capture the space of real-world object reflectance and a flexible illumination model that can represent real- world illumination with priors that combat the deleterious effects of image formation.
...
1
2
3
4
5
...