Virtual light transport matrices for non-line-of-sight imaging

  title={Virtual light transport matrices for non-line-of-sight imaging},
  author={Julio Marco and Adri{\'a}n Jarabo and Ji Hyun Nam and Xiaochun Liu and Miguel 'Angel Cosculluela and Andreas Velten and Diego Gutierrez},
  journal={2021 IEEE/CVF International Conference on Computer Vision (ICCV)},
The light transport matrix (LTM) is an instrumental tool in line-of-sight (LOS) imaging, describing how light interacts with the scene and enabling applications such as relighting or separation of illumination components. We introduce a framework to estimate the LTM of non-line-of-sight (NLOS) scenarios, coupling recent virtual forward light propagation models for NLOS imaging with the LOS light transport equation. We design computational projector-camera setups, and use these virtual imaging… 

Figures and Tables from this paper

Physics to the Rescue: Deep Non-line-of-sight Reconstruction for High-speed Imaging
A novel deep model is presented that incorporates the complementary physics priors of wave propagation and volume rendering into a neural network for high-quality and robust NLOS reconstruction, resulting in a deep model that generalizes well on real captures despite being exclusively trained on synthetic data.
Differentiable transient rendering
A novel differentiable transient rendering framework, and a novel correlated importance term, which links the time-integrated contribution of a path to its light throughput, and allows us to handle discontinuities in the light and sensor functions.


Non-line-of-sight Imaging with Partial Occluders and Surface Normals
This work introduces a factored NLOS light transport representation that accounts for partial occlusions and surface normals and develops a factorization approach for inverse time-resolved light transport and demonstrates high-fidelity NLOS reconstructions for challenging scenes both in simulation and with an experimental NLOS imaging system.
On the Effect of Reflectance on Phasor Field Non-Line-of-Sight Imaging
This work experimentally studies the reconstruction of hidden scenes composed of objects with increasingly specular materials, and empirically analyzed the effect of reflectance of the hidden scenes on reconstruction, to characterize the imaging capabilities of this promising new framework, and foster new NLOS imaging modalities.
Non-Line-of-Sight Surface Reconstruction Using the Directional Light-Cone Transform
It is demonstrated that surfaces fitted non-parametrically using the recovered normals are more accurate than those produced with NLOS surface reconstruction methods recently proposed, and are 1,000 times faster to compute than using inverse rendering.
Non-line-of-sight imaging using phasor-field virtual wave optics
This work shows that the problem of non-line-of-sight imaging can be formulated as one of diffractive wave propagation, by introducing a virtual wave field that is term the phasor field, and yields a new class of imaging algorithms that mimic the capabilities of line- of-sight cameras.
Confocal non-line-of-sight imaging based on the light-cone transform
The resolution bounds of NLOS imaging are quantified, its potential for real-time tracking is demonstrated, and efficient algorithms that incorporate image priors and a physically accurate noise model are derived.
Optical computing for fast light transport analysis
The experiments suggest that good quality relighting and transport inversion are possible from a few dozen low-dynamic range photos, even for scenes with complex shadows, caustics, and other challenging lighting effects.
Temporal frequency probing for 5D transient analysis of global light transport
It is observed that transient light transport is always separable in the temporal frequency domain, which makes it possible to analyze transient transport one temporal frequency at a time by trivially adapting techniques from conventional projector-to-camera transport.
Phasor field diffraction based reconstruction for fast non-line-of-sight imaging systems
The authors present an algorithm for non-line-of-sight imaging that is low complexity and allows fast and efficient reconstruction on a standard computer that will enable real-time NLOS imaging when used with emerging single-photon avalanche diode array detectors with resolution only limited by the temporal resolution of the sensor.
Non-line-of-sight Reconstruction Using Efficient Transient Rendering
This article introduces a novel reconstruction scheme that, by design, produces solutions that are consistent with state-of-the-art physically based rendering and evaluation on a variety of synthetic and experimental input data shows that it gracefully handles uncooperative scenes with high levels of noise or non-diffuse material reflectance.
Convolutional Approximations to the General Non-Line-of-Sight Imaging Operator
This work introduces a computationally tractable framework for solving the ellipsoidal tomography problem by using efficient deconvolution procedures to solve a linear least-squares problem involving the Gram operator.