# $\phi$-SfT: Shape-from-Template with a Physics-Based Deformation Model

@article{Kairanda2022phiSfTSW,
title={\$\phi\$-SfT: Shape-from-Template with a Physics-Based Deformation Model},
author={Navami Kairanda and Edith Tretschk and Mohamed A. Elgharib and Christian Theobalt and Vladislav Golyanik},
journal={2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2022},
pages={3938-3948}
}
• Published 22 March 2022
• Materials Science
• 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Shape-from-Template (SfT) methods estimate 3D surface deformations from a single monocular RGB camera while assuming a 3D state known in advance (a template). This is an important yet challenging problem due to the under-constrained nature of the monocular setting. Existing SfT techniques predominantly use geometric and simplified deformation models, which often limits their reconstruction abilities. In contrast to previous works, this paper proposes a new SfT approach explaining 2D…

## Figures and Tables from this paper

• Computer Science
ArXiv
• 2022
This survey focuses on state-of-the-art methods for dense non-rigid 3D reconstruction of various deformable objects and composite scenes from monocular videos or sets of monocular views.
• Mathematics
ArXiv
• 2022
Estimating the pose of an object from a monocular image is an inverse problem fundamental in computer vision. The ill-posed nature of this problem requires incorporating deformation priors to solve
• Computer Science
2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
• 2023
This work proposes to combine neural implicit representations for appearance modeling with neural ordinary differential equations (ODEs) for modelling planar physical phenomena to obtain a dynamic scene representation that can be identified directly from visual observations.
• Computer Science
ArXiv
• 2022
This paper proposes the first approach for 3D-consistent, dense and photorealistic novel view synthesis using just a single colour event stream as input and presents a neural radiance trained en-tirely in a self-supervised manner from events while pre-serving the original resolution of the colour event channels.
• Computer Science
ArXiv
• 2022
Ub4D handles large deformations, performs shape completion in occluded regions, and can operate on monocular RGB videos directly by using differentiable volume rendering, and includes a novel dynamic scene loss that enables the reconstruction of larger deformations by leveraging the coarse estimates of other methods.

## References

SHOWING 1-10 OF 62 REFERENCES

• Computer Science
2015 IEEE International Conference on Computer Vision (ICCV)
• 2015
Inspired by mesh-editing techniques, an As-Rigid-As-Possible (ARAP) deformation model that softly imposes local rigidity is used as a formalised isometric SfT as a constrained variational optimisation problem which is solved using iterative optimisation.
• Mathematics
IEEE Transactions on Pattern Analysis and Machine Intelligence
• 2015
It is established that isometric surfaces can be reconstructed unambiguously and that conformal surfacesCan be reconstructed up to a few discrete ambiguities and a global scale and the first algorithms to solve for the exact constraints are obtained analytically.
• Computer Science
2015 IEEE International Conference on Computer Vision (ICCV)
• 2015
This paper first compute a dense 3D template of the shape of the object, using a short rigid sequence, and subsequently perform online reconstruction of the non-rigid mesh as it evolves over time, which minimizes a robust photometric cost.
• Computer Science
2008 IEEE Conference on Computer Vision and Pattern Recognition
• 2008
This paper proposes an approach to learning shape priors, and uses a texture-based approach, to learn local deformation models, and combine them to reconstruct surfaces of arbitrary global shapes, and shows that their models are effective to reconstruct from single videos poorly-textured surfaces of arbitrarily shape.
• Computer Science
International Journal of Computer Vision
• 2010
A monocular 3D reconstruction algorithm for inextensible deformable surfaces that uses point correspondences between a single image of the deformed surface taken by a camera with known intrinsic parameters and a template to recover the 3D surface shape as seen in the image.
• Computer Science
2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 2020
A novel Lagrangian-Eulerian optimization formulation is introduced, including a cost function that penalizes differences to observations during an optimization run, which matches correspondence-free, sparse observations from a single-view depth image with a finite element simulation of deformable bodies.
• Computer Science
EuroVR
• 2018
This work proposes a new hybrid approach for monocular non-rigid reconstruction which it calls Hybrid Deformation Model Network (HDM-Net), which is learned by a deep neural network, with a combination of domain-specific loss functions.
• Computer Science
2015 IEEE International Conference on Computer Vision (ICCV)
• 2015
This paper addresses the problem of simultaneously recovering the 3D shape and pose of a deformable and potentially elastic object from 2D motion using an Expectation Maximization strategy, where each of these parameters are successively learned within partial M-steps, while robustly dealing with missing observations.
• Computer Science
2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
• 2018
A geometry-aware deep architecture that tackles the problem as usually done in analytic solutions: first perform 2D detection of the mesh and then estimate a 3D shape that is geometrically consistent with the image with a significantly lower computational time.
• Computer Science
Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662)
• 2000
This paper proposes a novel technique based on a non-rigid model, where the 3D shape in each frame is a linear combination of a set of basis shapes, and can be factored in a three-step process to yield pose, configuration and shape.