Deep Non-Rigid Structure From Motion

@article{Kong2019DeepNS,
  title={Deep Non-Rigid Structure From Motion},
  author={Chen Kong and Simon Lucey},
  journal={2019 IEEE/CVF International Conference on Computer Vision (ICCV)},
  year={2019},
  pages={1558-1567}
}
  • Chen Kong, S. Lucey
  • Published 30 July 2019
  • Computer Science
  • 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
Current non-rigid structure from motion (NRSfM) algorithms are mainly limited with respect to: (i) the number of images, and (ii) the type of shape variability they can handle. This has hampered the practical utility of NRSfM for many applications within vision. In this paper we propose a novel deep neural network to recover camera poses and 3D points solely from an ensemble of 2D image coordinates. The proposed neural network is mathematically interpretable as a multi-layer block sparse… Expand
Neural Dense Non-Rigid Structure from Motion with Latent Space Constraints
TLDR
This work introduces the first dense neural non-rigid structure from motion (N-NRSfM) approach, which can be trained end-to-end in an unsupervised manner from 2D point tracks and enables multiple applications including shape compression, completion and interpolation, among others. Expand
Deep NRSfM++: Towards 3D Reconstruction in the Wild
TLDR
Deep NRSfM++ is presented, an approach to allow NRSfFM to be truly applicable in the wild by offering up innovative solutions to missing/occluded points and weak-perspective camera models. Expand
Procrustean Regression Networks: Learning 3D Structure of Non-Rigid Objects from 2D Annotations
TLDR
A novel framework for training neural networks which is capable of learning 3D information of non-rigid objects when only 2D annotations are available as ground truths and shows superior reconstruction performance to the state-of-the-art method on the Human 3.6M, 300-VW, and SURREAL datasets, even though the underlying network structure is very simple. Expand
PAUL: Procrustean Autoencoder for Unsupervised Lifting
  • Chaoyang Wang, S. Lucey
  • Computer Science
  • 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2021
TLDR
This paper advocates for a 3D deep auto-encoder framework to be used explicitly as the NRSfM prior, and refers to this architecture as a Procustean Autoencoder for Unsupervised Lifting (PAUL), and demonstrates state-of-the-art performance across a number of benchmarks. Expand
Unsupervised 3D Reconstruction and Grouping of Rigid and Non-Rigid Categories.
  • Antonio Agudo
  • Computer Science, Medicine
  • IEEE transactions on pattern analysis and machine intelligence
  • 2020
In this paper we present an approach to jointly recover camera pose, 3D shape, and object and deformation type grouping, from incomplete 2D annotations in a multi-instance collection of RGB images.Expand
When to Use Convolutional Neural Networks for Inverse Problems
TLDR
This work argues that for some types of inverse problems the CNN approximation breaks down leading to poor performance, and argues that the CSC approach should be used instead and validate this argument with empirical evidence. Expand
LASR: Learning Articulated Shape Reconstruction from a Monocular Video
TLDR
This work introduces a template-free approach to learn 3D shapes from a single video with an analysis-by-synthesis strategy that forward-renders object silhouette, optical flow, and pixel values to compare with video observations, which generates gradients to adjust the camera, shape and motion parameters. Expand
Shelf-Supervised Mesh Prediction in the Wild
TLDR
This work proposes a learning-based approach that can train from unstructured image collections, supervised by only segmentation outputs from off-the-shelf recognition systems (i.e. ‘shelf-supervised’). Expand
Deep Facial Non-Rigid Multi-View Stereo
TLDR
This method optimizes the 3D face shape by explicitly enforcing multi-view appearance consistency, which is known to be effective in recovering shape details according to conventional multi- view stereo methods. Expand
To The Point: Correspondence-driven monocular 3D category reconstruction
TLDR
To The Point (TTP), a method for reconstructing 3D objects from a single image using 2D to 3D correspondences learned from weak supervision, uses a simple per-sample optimization problem to replace CNN-based regression of camera pose and non-rigid deformation and thereby obtain substantially more accurate 3D reconstructions. Expand
...
1
2
3
...

References

SHOWING 1-10 OF 67 REFERENCES
Deep Interpretable Non-Rigid Structure from Motion
TLDR
A novel deep neural network to recover camera poses and 3D points solely from an ensemble of 2D image coordinates is proposed, which is mathematically interpretable as a multi-layer block sparse dictionary learning problem, and can handle problems of unprecedented scale and shape complexity. Expand
Scalable Dense Non-rigid Structure-from-Motion: A Grassmannian Perspective
TLDR
This paper proposes a new approach for dense NRSfM by modeling the problem on a Grassmann manifold that is scalable and more accurate than state-of-the-art methods, and more robust to noise and generalizes to highly nonlinear deformations. Expand
Prior-Less Compressible Structure from Motion
  • Chen Kong, S. Lucey
  • Mathematics, Computer Science
  • 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2016
TLDR
It is demonstrated how the problem of SfM - assuming compressible 3D structures - can be theoretically characterized as a block sparse dictionary learning problem. Expand
Structure from Category: A Generic and Prior-Less Approach
TLDR
This paper proposes the concept of Structure from Category (SfC) to reconstruct 3D structure of generic objects solely from images with no shape and motion constraint (i.e. prior-less), and demonstrates the superiority of the approach to the state-of-the-art low-rank NRSfM approaches. Expand
Re-thinking non-rigid structure from motion
TLDR
This work presents a novel approach to non-rigid structure from motion (NRSFM) from an orthographic video sequence, based on a new interpretation of the problem, which only assumes that small neighborhoods of shapes are well-modeled with a linear subspace. Expand
Learning Spatially-Smooth Mappings in Non-Rigid Structure From Motion
TLDR
A new NRSFM method is proposed that addresses deformations as spatial variations in shape space and then enforcing spatial, rather than temporal, smoothness by modeling each 3D shape coefficient as a function of its input 2D shape. Expand
Kernel non-rigid structure from motion
TLDR
This work demonstrates how the kernel trick can be applied in standard NRSFM and model complex, deformable 3D shapes as the outputs of a non-linear mapping whose inputs are points within a low-dimensional shape space. Expand
Complex Non-rigid Motion 3D Reconstruction by Union of Subspaces
TLDR
It is demonstrated empirically that the current state of the art approach to NRSfM exhibits poor reconstruction performance on complex motion and proposed that this limitation can be circumvented by modeling complex motion as a union of subspaces. Expand
A Benchmark and Evaluation of Non-Rigid Structure from Motion
TLDR
A benchmark and a scrupulous evaluation using this data set, which evaluates 18 different methods with available code that reasonably spans the state of the art in sparse nrsfm, will provide benchmark tools for further development in this challenging field. Expand
Convolutional Sparse Coding for Trajectory Reconstruction
  • Yingying Zhu, S. Lucey
  • Mathematics, Medicine
  • IEEE Transactions on Pattern Analysis and Machine Intelligence
  • 2015
TLDR
This paper draws upon a well known result centering around the Reduced Isometry Property (RIP) condition for sparse signal reconstruction to relax the requirement that the full trajectory basis composed with the camera matrix must be well conditioned. Expand
...
1
2
3
4
5
...