A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation

@article{Mayer2016ALD,
  title={A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation},
  author={Nikolaus Mayer and Eddy Ilg and Philip H{\"a}usser and Philipp Fischer and Daniel Cremers and Alexey Dosovitskiy and Thomas Brox},
  journal={2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2016},
  pages={4040-4048}
}
  • N. Mayer, Eddy Ilg, T. Brox
  • Published 7 December 2015
  • Computer Science
  • 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Recent work has shown that optical flow estimation can be formulated as a supervised learning task and can be successfully solved with convolutional networks. [] Key Method Our datasets are the first large-scale datasets to enable training and evaluation of scene flow methods. Besides the datasets, we present a convolutional network for real-time disparity estimation that provides state-of-the-art results. By combining a flow and disparity estimation network and training it jointly, we demonstrate the first…
Finding Correspondences for Optical Flow and Disparity Estimations using a Sub-pixel Convolution-based Encoder-Decoder Network
TLDR
A novel sub- pixel convolution-based encoder-decoder network for optical flow and disparity estimations, which can extend FlowNetS and DispNet by replacing the deconvolution layers with sup-pixel convolution blocks.
SceneEDNet: A Deep Learning Approach for Scene Flow Estimation
TLDR
This paper introduces a first effort to apply a deep learning method for direct estimation of scene flow by presenting a fully convolutional neural network with an encoder-decoder (ED) architecture.
Unsupervised Deep Learning for Optical Flow Estimation
TLDR
This work devise a simple yet effective unsupervised method for learning optical flow, by directly minimizing photometric consistency by using image warping by the estimated flow.
A Tiny Diagnostic Dataset and Diverse Modules for Learning-Based Optical Flow Estimation
TLDR
A tiny diagnostic dataset called FlowClevr is proposed to quickly evaluate various modules that can use to enhance standard CNN architectures and finds that a deformable module can improve model prediction accuracy by around 30% to 100% in most tasks and more significantly reduce boundary blur.
Loop-Net: Joint Unsupervised Disparity and Optical Flow Estimation of Stereo Videos With Spatiotemporal Loop Consistency
TLDR
This letter proposes a joint framework that estimates disparity and optical flow of stereo videos and generalizes across various video frames by considering the spatiotemporal relation between the disparity and flow without supervision and introduces a video-based training scheme using the c-LSTM to reinforce the temporal consistency.
Learning optical flow from still images
TLDR
This paper introduces a framework to generate accurate ground-truth optical flow annotations quickly and in large amounts from any readily available single real picture, using an off-the-shelf monocular depth estimation network to build a plausible point cloud for the observed scene.
Estimating optical flow with convolutional neural networks
TLDR
This thesis presents an orthogonal approach to estimate optical flow with Convolutional Neural Networks and shows that such networks are able to learn a better heuristic than engineered methods and are state of the art in motion boundary and occlusion estimation.
Self-Supervised Monocular Scene Flow Estimation
  • Junhwa Hur, S. Roth
  • Computer Science
    2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2020
TLDR
This work designs a single convolutional neural network (CNN) that successfully estimates depth and 3D motion simultaneously from a classical optical flow cost volume, and adopts self-supervised learning with 3D loss functions and occlusion reasoning to leverage unlabeled data.
Unsupervised Learning for Depth, Ego-Motion, and Optical Flow Estimation Using Coupled Consistency Conditions
TLDR
This paper proposes a framework that trains networks while using a different type of data with combined losses that are derived from a coupled consistency structure, and shows that the view-synthesis-based photometric loss enhances the depth and ego-motion accuracy via scene projection.
Temporal Interpolation as an Unsupervised Pretraining Task for Optical Flow Estimation
TLDR
This work investigates frame interpolation as a proxy task for optical flow using real movies and trains a CNN unsupervised for temporal interpolation, which outperforms similar architectures that were trained supervised using synthetic optical flow.
...
...

References

SHOWING 1-10 OF 38 REFERENCES
FlowNet: Learning Optical Flow with Convolutional Networks
TLDR
This paper constructs CNNs which are capable of solving the optical flow estimation problem as a supervised learning task, and proposes and compares two architectures: a generic architecture and another one including a layer that correlates feature vectors at different image locations.
Stereoscopic Scene Flow Computation for 3D Motion Understanding
TLDR
A variational framework for the estimation of stereoscopic scene flow, i.e., the motion of points in the three-dimensional world from stereo image sequences, which partially decouple the depth estimation from the motion estimation, which has many practical advantages.
A Variational Method for Scene Flow Estimation from Stereo Sequences
TLDR
This paper proposes to recover the scene flow by coupling the optical flow estimation in both cameras with dense stereo matching between the images, thus reducing the number of unknowns per image point.
Scene flow estimation by growing correspondence seeds
TLDR
A simple seed growing algorithm for estimating scene flow in a stereo setup that is accurate for complex scenes with large motions and produces temporally-coherent stereo disparity and optical flow results is presented.
A Database and Evaluation Methodology for Optical Flow
TLDR
This paper proposes a new set of benchmarks and evaluation methods for the next generation of optical flow algorithms and analyzes the results obtained to date to draw a large number of conclusions.
Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
TLDR
This paper employs two deep network stacks: one that makes a coarse global prediction based on the entire image, and another that refines this prediction locally, and applies a scale-invariant error to help measure depth relations rather than scale.
Piecewise Rigid Scene Flow
TLDR
A novel model that represents the dynamic 3D scene by a collection of planar, rigidly moving, local segments is introduced that achieves leading performance levels, exceeding competing3D scene flow methods, and even yielding better 2D motion estimates than all tested dedicated optical flow techniques.
Object scene flow for autonomous vehicles
TLDR
A novel model and dataset for 3D scene flow estimation with an application to autonomous driving by representing each element in the scene by its rigid motion parameters and each superpixel by a 3D plane as well as an index to the corresponding object.
Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches
TLDR
This work presents a method for extracting depth information from a rectified image pair by learning a similarity measure on small image patches using a convolutional neural network and examines two network architectures for this task: one tuned for speed, the other for accuracy.
3D Scene Flow Estimation with a Piecewise Rigid Scene Model
TLDR
This work proposes to represent the dynamic scene as a collection of rigidly moving planes, into which the input images are segmented, and shows that a view-consistent multi-frame scheme significantly improves accuracy, especially in the presence of occlusions, and increases robustness against adverse imaging conditions.
...
...