NRSfM-Flow: Recovering Non-Rigid Scene Flow from Monocular Image Sequences

  title={NRSfM-Flow: Recovering Non-Rigid Scene Flow from Monocular Image Sequences},
  author={Vladislav Golyanik and Aman Shankar Mathur and Didier Stricker},
Scene flow recovery from monocular image sequences is an emerging field in computer vision. While existing Monocular Scene Flow (MSF) methods extend the classical optical flow formulation to estimate depths/disparities and 3D motion, we propose a framework based on Non-Rigid Structure from Motion (NRSfM) technique — NRSfM-Flow. Therefore, both problems are formulated in the continuous domain and relation between them is established. To cope with real data, we propose two preprocessing steps for… Expand
Introduction to Coherent Depth Fields for Dense Monocular Surface Recovery
It is shown experimentally that CDF achieves state-of-the-art in dense NRSfM including scenarios with long and large occlusions, inaccurate correspondences as well as inaccurate initialisations, without requiring any additional pre-processing steps. Expand
HDM-Net: Monocular Non-Rigid 3D Reconstruction with Learned Deformation Model
This work proposes a new hybrid approach for monocular non-rigid reconstruction which it calls Hybrid Deformation Model Network (HDM-Net), which is learned by a deep neural network, with a combination of domain-specific loss functions. Expand
Mono-SF: Multi-View Geometry Meets Single-View Depth for Monocular Scene Flow Estimation of Dynamic Traffic Scenes
A novel monocular 3D scene flow estimation method, called Mono-SF, which jointly estimates the 3D structure and motion of the scene by combining multi-view geometry and single-view depth information. Expand
Proceedings of Seminar and Project 2D Image Processing and Augmented Reality
3D reconstruction of deformed objects from a single monocular 2D image is still a challenging topic.Current methods either require reference template frame, dense correspondences, or tracking points.Expand
SceneEDNet: A Deep Learning Approach for Scene Flow Estimation
This paper introduces a first effort to apply a deep learning method for direct estimation of scene flow by presenting a fully convolutional neural network with an encoder-decoder (ED) architecture. Expand
Scalable Dense Monocular Surface Reconstruction
The proposed Scalable Monocular Surface Reconstruction (SMSR) combines strengths of several algorithms, i.e., it is scalable with the number of points, can handle sparse and dense settings as well as different types of motions and deformations. Expand
Neural Dense Non-Rigid Structure from Motion with Latent Space Constraints
This work introduces the first dense neural non-rigid structure from motion (N-NRSfM) approach, which can be trained end-to-end in an unsupervised manner from 2D point tracks and enables multiple applications including shape compression, completion and interpolation, among others. Expand
Dense Batch Non-Rigid Structure from Motion in a Second
The modification results in an efficient single-core CPU implementation enabling dense factorisations of long image sequences with tens of thousands of points into camera pose and non-rigid shape in seconds, i.e., at least two orders of magnitude faster than the runtimes reported in the literature so far. Expand
DeepFaceFlow: In-the-Wild Dense 3D Facial Motion Estimation
This work proposes DeepFaceFlow, a robust, fast, and highly-accurate framework for the dense estimation of 3D non-rigid facial flow between pairs of monocular images, and incorporates its framework in a full-head state-of-the-art facial video synthesis method. Expand
Robust Methods for Dense Monocular Non-Rigid 3D Reconstruction and Alignment of Point Clouds
XXI Zusammenfassung XXIII


Monocular scene flow estimation via variational method
A monocular camera scene flow estimation algorithm that can use three sequential frames at least to get scene flow in world coordinate, without optical flow or disparity inputting is proposed. Expand
Monocular Concurrent Recovery of Structure and Motion Scene Flow
A variational method of joint three-dimensional structure and motion scene flow recovery from a single image sequence by minimizing a functional with a term of conformity of scene flow and depth to the image sequence spatiotemporal variations, and quadratic smoothness regularization terms. Expand
Video Pop-up: Monocular 3D Reconstruction of Dynamic Scenes
An unsupervised approach to the challenging problem of simultaneously segmenting the scene into its constituent objects and reconstructing a 3D model of the scene and evaluating the motion segmentation functionality of the approach on the Berkeley Motion Segmentation Dataset. Expand
Depth and Scene Flow from a Single Moving Camera
We show that it is possible to reconstruct scene flow and depth from a single moving camera whose motion is known. To do so, we assume that the local scene motion can be approximated by a constantExpand
Occlusion-aware video registration for highly non-rigid objects
This paper addresses the problem of video registration for dense non-rigid structure from motion under suboptimal conditions, such as noise, self-occlusions, considerable external occlusions or specularities, by improving previously proposed variational optimization techniques for multi-frame optical flow through detection, tracking and handling of uncertain flow field estimates. Expand
Basis constrained 3D scene flow on a dynamic proxy
This work proposes a variational formulation of scene flow relative to a coarse proxy geometry, which is better suited for many views and takes known proxy motion into account, which enables 3D trajectory reconstruction when only a single view is available. Expand
Shape and motion from image streams under orthography: a factorization method
A factorization method is developed that can overcome the difficulty by recovering shape and motion under orthography without computing depth as an intermediate step, and gives accurate results. Expand
Dense Variational Reconstruction of Non-rigid Surfaces from Monocular Video
This paper offers the first variational approach to the problem of dense 3D reconstruction of non-rigid surfaces from a monocular video sequence and reconstructs highly deforming smooth surfaces densely and accurately directly from video, without the need for any prior models or shape templates. Expand
A Database and Evaluation Methodology for Optical Flow
This paper proposes a new set of benchmarks and evaluation methods for the next generation of optical flow algorithms and analyzes the results obtained to date to draw a large number of conclusions. Expand
Joint Estimation of Motion, Structure and Geometry from Stereo Sequences
A joint energy functional is proposed that integrates spatial and temporal information from two subsequent image pairs subject to an unknown stereo setup and a normalisation of image and stereo constraints such that deviations from model assumptions can be interpreted in a geometrical way. Expand