$\phi$-SfT: Shape-from-Template with a Physics-Based Deformation Model

@article{Kairanda2022phiSfTSW,
  title={\$\phi\$-SfT: Shape-from-Template with a Physics-Based Deformation Model},
  author={Navami Kairanda and Edith Tretschk and Mohamed A. Elgharib and Christian Theobalt and Vladislav Golyanik},
  journal={2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2022},
  pages={3938-3948}
}
Shape-from-Template (SfT) methods estimate 3D surface deformations from a single monocular RGB camera while assuming a 3D state known in advance (a template). This is an important yet challenging problem due to the under-constrained nature of the monocular setting. Existing SfT techniques predominantly use geometric and simplified deformation models, which often limits their reconstruction abilities. In contrast to previous works, this paper proposes a new SfT approach explaining 2D… 
[PDF] Semantic Reader
6 Citations
Highly Influential Citations
1
Background Citations
4
Methods Citations
2

Figures and Tables from this paper

6 Citations

State of the Art in Dense Monocular Non-Rigid 3D Reconstruction

This survey focuses on state-of-the-art methods for dense non-rigid 3D reconstruction of various deformable objects and composite scenes from monocular videos or sets of monocular views.

Deformable Surface Reconstruction via Riemannian Metric Preservation

Estimating the pose of an object from a monocular image is an inverse problem fundamental in computer vision. The ill-posed nature of this problem requires incorporating deformation priors to solve…

Neural Implicit Representations for Physical Parameter Inference from a Single Video

This work proposes to combine neural implicit representations for appearance modeling with neural ordinary differential equations (ODEs) for modelling planar physical phenomena to obtain a dynamic scene representation that can be identified directly from visual observations.

EventNeRF: Neural Radiance Fields from a Single Colour Event Camera

This paper proposes the first approach for 3D-consistent, dense and photorealistic novel view synthesis using just a single colour event stream as input and presents a neural radiance trained en-tirely in a self-supervised manner from events while pre-serving the original resolution of the colour event channels.

Fast Non-Rigid Radiance Fields from Monocularized Data

i.e

Unbiased 4D: Monocular 4D Reconstruction with a Neural Deformation Model

Ub4D handles large deformations, performs shape completion in occluded regions, and can operate on monocular RGB videos directly by using differentiable volume rendering, and includes a novel dynamic scene loss that enables the reconstruction of larger deformations by leveraging the coarse estimates of other methods.

References

SHOWING 1-10 OF 62 REFERENCES

As-Rigid-as-Possible Volumetric Shape-from-Template

Inspired by mesh-editing techniques, an As-Rigid-As-Possible (ARAP) deformation model that softly imposes local rigidity is used as a formalised isometric SfT as a constrained variational optimisation problem which is solved using iterative optimisation.

Shape-from-Template

It is established that isometric surfaces can be reconstructed unambiguously and that conformal surfacesCan be reconstructed up to a few discrete ambiguities and a global scale and the first algorithms to solve for the exact constraints are obtained analytically.

Direct, Dense, and Deformable: Template-Based Non-rigid 3D Reconstruction from RGB Video

This paper first compute a dense 3D template of the shape of the object, using a short rigid sequence, and subsequently perform online reconstruction of the non-rigid mesh as it evolves over time, which minimizes a robust photometric cost.

Local deformation models for monocular 3D shape recovery

This paper proposes an approach to learning shape priors, and uses a texture-based approach, to learn local deformation models, and combine them to reconstruct surfaces of arbitrary global shapes, and shows that their models are effective to reconstruct from single videos poorly-textured surfaces of arbitrarily shape.

Monocular Template-based Reconstruction of Inextensible Surfaces

A monocular 3D reconstruction algorithm for inextensible deformable surfaces that uses point correspondences between a single image of the deformed surface taken by a camera with known intrinsic parameters and a template to recover the 3D surface shape as seen in the image.

Correspondence-Free Material Reconstruction using Sparse Surface Constraints

A novel Lagrangian-Eulerian optimization formulation is introduced, including a cost function that penalizes differences to observations during an optimization run, which matches correspondence-free, sparse observations from a single-view depth image with a finite element simulation of deformable bodies.

HDM-Net: Monocular Non-Rigid 3D Reconstruction with Learned Deformation Model

This work proposes a new hybrid approach for monocular non-rigid reconstruction which it calls Hybrid Deformation Model Network (HDM-Net), which is learned by a deep neural network, with a combination of domain-specific loss functions.

Learning Shape, Motion and Elastic Models in Force Space

This paper addresses the problem of simultaneously recovering the 3D shape and pose of a deformable and potentially elastic object from 2D motion using an Expectation Maximization strategy, where each of these parameters are successively learned within partial M-steps, while robustly dealing with missing observations.

Geometry-Aware Network for Non-rigid Shape Prediction from a Single View

A geometry-aware deep architecture that tackles the problem as usually done in analytic solutions: first perform 2D detection of the mesh and then estimate a 3D shape that is geometrically consistent with the image with a significantly lower computational time.

Recovering non-rigid 3D shape from image streams

This paper proposes a novel technique based on a non-rigid model, where the 3D shape in each frame is a linear combination of a set of basis shapes, and can be factored in a three-step process to yield pose, configuration and shape.
...