Efficient and Differentiable Shadow Computation for Inverse Problems

@article{Lyu2021EfficientAD,
  title={Efficient and Differentiable Shadow Computation for Inverse Problems},
  author={Linjie Lyu and Marc Habermann and Lingjie Liu and R. MallikarjunB. and Ayush Tewari and Christian Theobalt},
  journal={2021 IEEE/CVF International Conference on Computer Vision (ICCV)},
  year={2021},
  pages={13087-13096}
}
Differentiable rendering has received increasing interest for image-based inverse problems. It can benefit traditional optimization-based solutions to inverse problems, but also allows for self-supervision of learning-based approaches for which training data with ground truth annotation is hard to obtain. However, existing differentiable renderers either do not model visibility of the light sources from the different points in the scene, responsible for shadows in the images, or are very slow… 

Towards High Fidelity Monocular Face Reconstruction with Rich Reflectance using Self-supervised Learning and Ray Tracing

This paper proposes a new method that greatly improves reconstruction quality and robustness in general scenes and achieves this by combining a CNN encoder with a differentiable ray tracer, which enables to take a big leap forward in reconstruction quality of shape, appearance and lighting even in scenes with difficult illumination.

Neural Radiance Transfer Fields for Relightable Novel-view Synthesis with Global Illumination

This work proposes a method for scene relighting under novel views by learning a neural precomputed radiance transfer function, which implicitly handles global illumination effects using novel environment maps, and can be solely supervised on a set of real images of the scene under a single unknown lighting condition.

Advances in Neural Rendering

This state‐of‐the‐art report on advances in neural rendering focuses on methods that combine classical rendering principles with learned 3D scene representations, often now referred to as neural scene representations.

DeepShadow: Neural Shape from Shadow

This paper presents ‘DeepShadow’, a one-shot method for recovering the depth map and surface normals from photometric stereo shadow maps, which is the first to reconstruct 3D shape-from-shadows using neural networks.

Recursive analytic spherical harmonics gradient for spherical lights

This paper develops analytical recursive formulae to compute the spatial gradients of SH coefficients for spherical light and integrates this algorithm in a shading system able to render fully dynamic scenes with several hundreds of spherical lights in real time.

Face Relighting with Geometrically Consistent Shadows

This work proposes a novel differentiable algorithm for synthesizing hard shadows based on ray tracing, which it incorporates into training the face relighting model and demonstrates that this differentiable hard shadow modeling improves the quality of the estimated face geometry over diffuse shading models.

S2F2: Self-Supervised High Fidelity Face Reconstruction from Monocular Image

This work achieves, for the first time, high fidelity face reconstruction using self-supervised learning only, and allows it to solve the challenging problem of decoupling face reflectance from geometry using a single image, at high computational speed.

HDHumans: A Hybrid Approach for High-fidelity Digital Humans

This work proposes HDHumans, which is the first method for HD human character synthesis that jointly produces an accurate and temporally coherent 3D deforming surface and highly photo-realistic images of arbitrary novel views and of motions not.

Plateau-free Differentiable Path Tracing

This work describes two Monte Carlo estimators to compute plateau-free gradients efficiently, i.e., with low variance, and shows that these translate into net-gains in optimization error and runtime performance.

References

SHOWING 1-10 OF 46 REFERENCES

Differentiable Monte Carlo ray tracing through edge sampling

This work introduces a general-purpose differentiable ray tracer, which is the first comprehensive solution that is able to compute derivatives of scalar functions over a rendered image with respect to arbitrary scene parameters such as camera pose, scene geometry, materials, and lighting parameters.

Path-space differentiable rendering

It is shown how path integrals can be differentiated with respect to arbitrary differentiable changes of a scene and the design of new Monte Carlo estimators that offer significantly better efficiency than state-of-the-art methods in handling complex geometric discontinuities and light transport phenomena such as caustics are considered.

Physics-based differentiable rendering: from theory to implementation

Differentiable renderers allow "rendering losses" to be computed with complex light transport effects captured and can be used as generative models that synthesize photorealistic images, incorporated into probabilistic inference and machine learning pipelines.

Inverse Path Tracing for Joint Material and Lighting Estimation

A novel optimization method using a differentiable Monte Carlo renderer that computes derivatives with respect to the estimated unknown illumination and material properties enables joint optimization for physically correct light transport and material models using a tailored stochastic gradient descent.

A Versatile Scene Model with Differentiable Visibility Applied to Generative Pose Estimation

A new scene representation is presented that enables an analytically differentiable closed-form formulation of surface visibility and results in a new image formation model that represents opaque objects by a translucent medium with a smooth Gaussian density distribution which turns visibility into a smooth phenomenon.

Soft Rasterizer: A Differentiable Renderer for Image-Based 3D Reasoning

This work proposes a truly differentiable rendering framework that is able to directly render colorized mesh using differentiable functions and back-propagate efficient supervision signals to mesh vertices and their attributes from various forms of image representations, including silhouette, shading and color images.

A differential theory of radiative transfer

A differential theory of radiativeTransfer is introduced, which shows how individual components of the radiative transfer equation (RTE) can be differentiated with respect to arbitrary differentiable changes of a scene.

Deferred Neural Rendering: Image Synthesis using Neural Textures

This work proposes Neural Textures, which are learned feature maps that are trained as part of the scene capture process that can be utilized to coherently re-render or manipulate existing video content in both static and dynamic environments at real-time rates.

Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF From a Single Image

A deep inverse rendering framework for indoor scenes, which combines novel methods to map complex materials to existing indoor scene datasets and a new physically-based GPU renderer to create a large-scale, photorealistic indoor dataset.

Unbiased warped-area sampling for differentiable rendering

This work applies the divergence theorem to the derivative of the rendering integral to convert the boundary integral into an area integral, and develops an efficient Monte Carlo sampling algorithm for solving the area integral.