Neural Radiance Fields Approach to Deep Multi-View Photometric Stereo

@article{Kaya2021NeuralRF,
  title={Neural Radiance Fields Approach to Deep Multi-View Photometric Stereo},
  author={Berk Kaya and Suryansh Kumar and Francesco Sarno and Vittorio Ferrari and Luc Van Gool},
  journal={2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year={2021},
  pages={3967-3979}
}
  • Berk KayaSuryansh Kumar L. Gool
  • Published 11 October 2021
  • Computer Science
  • 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
We present a modern solution to the multi-view photometric stereo problem (MVPS). Our work suitably exploits the image formation model in a MVPS experimental setup to recover the dense 3D reconstruction of an object from images. We procure the surface orientation using a photometric stereo (PS) image formation model and blend it with a multi-view neural radiance field representation to recover the object’s surface geometry. Contrary to the previous multi-staged framework to MVPS, where the… 
[PDF] Semantic Reader
8 Citations
Highly Influential Citations
1
Background Citations
5
Methods Citations
4
Results Citations
2

Figures and Tables from this paper

8 Citations

PS-NeRF: Neural Inverse Rendering for Multi-view Photometric Stereo

This paper presents a neural inverse rendering method for MVPS based on implicit representation that achieves far more accurate shape reconstruction than existing MVPS and neural rendering methods.

Multi-View Photometric Stereo Revisited

A simple, practical approach to MVPS, which works well for isotropic as well as other object material types such as anisotropic and glossy, and introduces neural volume rendering methodology for a trustworthy fusion of MVS and PS measurements.

Uncertainty-Aware Deep Multi-View Photometric Stereo

This paper presents a simple and effective solution to the longstanding classical multi-view photometric stereo (MVPS) problem, and proposes to learn the implicit neural shape representation via a multilayer perceptron (MLP), providing superior dense surface reconstruction.

nLMVS-Net: Deep Non-Lambertian Multi-View Stereo

Extensive quantitative evaluations on newly established synthetic and real-world datasets show that nLMVS-Net can robustly and accurately recover the shape and reflectance of complex objects in natural settings.

Robustifying the Multi-Scale Representation of Neural Radiance Fields

A robust multi-scale neural radiance representation approach to simultaneously overcome both real-world imaging issues and NeRF-inspired approaches by leveraging the fundamentals of scene rigidity.

GeoNeRF: Generalizing NeRF with Geometry Priors

GeoNeRF is presented, a generalizable photorealistic novel view synthesis method based on neural radiance fields that outperforms state-of-the- art generalizable neural rendering models on various syn-thetic and real datasets.

Combining Photogrammetry and Photometric Stereo to Achieve Precise and Complete 3D Reconstruction

Image-based 3D reconstruction has been employed in industrial metrology for micro-measurements and quality control purposes. However, generating a highly-detailed and reliable 3D reconstruction of…

Compressing Explicit Voxel Grid Representations: fast NeRFs become also small

This work proposes Re:NeRF, an approach that specifically targets EVG-NeRFs compressibility, aiming to reduce memory storage of NeRF models while maintaining comparable performance.

References

SHOWING 1-10 OF 128 REFERENCES

A Differential Volumetric Approach to Multi-View Photometric Stereo

The key point of this method is the signed distance field parameterisation and its relation to the surface normal that is exploited in order to obtain a linear partial differential equation which is solved in a variational framework, that combines multiple images from multiple points of view in a single system.

Uncalibrated Neural Inverse Rendering for Photometric Stereo of General Surfaces

This paper presents an uncalibrated deep neural network framework for the photometric stereo problem and explicitly models the concave and convex parts of a complex surface to consider the effects of interreflections in the image formation process.

MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo

This work proposes a generic deep neural network that can reconstruct radiance fields from only three nearby input views via fast network inference, and leverages plane-swept cost volumes for geometry-aware scene reasoning, and combines this with physically based volume rendering for neural radiance field reconstruction.

Deep Photometric Stereo Network

A deep photometric stereo network (DPSN) that takes reflectance observations under varying light directions and infers the corresponding surface normal per pixel and Evaluation using simulation and real-world scenes shows effectiveness of the proposed approach over previous techniques.

CNN-PS: CNN-based Photometric Stereo for General Non-Convex Surfaces

The experimental results on both synthetic and real datasets show that the method outperforms conventional BRDF-based photometric stereo algorithms especially when scenes are highly non-convex.

Neural Inverse Rendering for General Reflectance Photometric Stereo

A physics based unsupervised learning framework where surface normals and BRDFs are predicted by the network and fed into the rendering equation to synthesize observed images, which is shown to achieve the state-of-the-art performance on a challenging real-world scene benchmark.

Deep Stereo Using Adaptive Thin Volume Representation With Uncertainty Awareness

  • Shuo ChengZexiang Xu Hao Su
  • Computer Science
    2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2020
The proposed ATV consists of only a small number of planes with low memory and computation costs; yet, it efficiently partitions local depth ranges within learned small uncertainty intervals, which enables reconstruction with high completeness and accuracy in a coarse-to-fine fashion.

D-NeRF: Neural Radiance Fields for Dynamic Scenes

D-NeRF is introduced, a method that extends neural radiance fields to a dynamic domain, allowing to reconstruct and render novel images of objects under rigid and non-rigid motions from a single camera moving around the scene.

Deep Photometric Stereo for Non-Lambertian Surfaces

This paper addresses the problem of photometric stereo, in both calibrated and uncalibrated scenarios, for non-Lambertian surfaces based on deep learning with a fully convolutional deep network, which is able to handle surfaces with general and unknown isotropic reflectance.

Multi-View Photometric Stereo: A Robust Solution and Benchmark Dataset for Spatially Varying Isotropic Materials

A method to capture both 3D shape and spatially varying reflectance with a multi-view photometric stereo technique that works for general isotropic materials and quantitatively evaluate state-of-the-art MVPS on a newly collected benchmark dataset is presented.
...