MirrorFlow: Exploiting Symmetries in Joint Optical Flow and Occlusion Estimation

@article{Hur2017MirrorFlowES,
  title={MirrorFlow: Exploiting Symmetries in Joint Optical Flow and Occlusion Estimation},
  author={Junhwa Hur and Stefan Roth},
  journal={2017 IEEE International Conference on Computer Vision (ICCV)},
  year={2017},
  pages={312-321}
}
  • Junhwa Hur, S. Roth
  • Published 17 August 2017
  • Computer Science
  • 2017 IEEE International Conference on Computer Vision (ICCV)
Optical flow estimation is one of the most studied problems in computer vision, yet recent benchmark datasets continue to reveal problem areas of today’s approaches. Occlusions have remained one of the key challenges. In this paper, we propose a symmetric optical flow method to address the well-known chicken-and-egg relation between optical flow and occlusions. In contrast to many state-ofthe- art methods that consider occlusions as outliers, possibly filtered out during post-processing, we… 
OIFlow: Occlusion-Inpainting Optical Flow Estimation by Unsupervised Learning
TLDR
OIFlow is presented, an occlusion-inpainting framework to make full use of occlusions regions, and a new appearance-flow network is proposed to inpaint occluded flows based on the image content.
Learning to Estimate Hidden Motions with Global Motion Aggregation
TLDR
This paper argues that the occlusion problem can be better solved in the two-frame case by modelling image self-similarities, and introduces a global motion aggregation module, a transformer-based approach to find long-range dependencies between pixels in the first image, and perform global aggregation on the corresponding motion features.
Self-Supervised Optical Flow Estimation by Projective Bootstrap
TLDR
This paper argues that optical flow can be cast as a geometrical warping between two successive video frames and devise a deep architecture to estimate such transformation in two stages, and shows how such global transformation can be approximated with a homography and extend spatial transformer layers so that they can be employed to compute the flow field implied by such transformation.
Structure-from-Motion-Aware PatchMatch for Adaptive Optical Flow Estimation
TLDR
This paper proposes a novel structure-from-motion-aware PatchMatch approach that, in contrast to existing matching techniques, combines two hierarchical feature matching methods: a recent two-framePatchMatch approach for optical flow estimation (general motion) and a specifically tailored three-frame Patch match approach for rigid scene reconstruction (SfM).
Continual Occlusion and Optical Flow Estimation
TLDR
It is shown that providing occlusion masks as an additional input to flow estimation improves the standard performance metric by more than 25% on both KITTI and Sintel, and a novel method for incorporating information from past frames into flow estimation is introduced.
Continual Occlusions and Optical Flow Estimation
TLDR
It is shown that providing occlusion masks as an additional input to flow estimation improves the standard performance metric by more than 25\% on both KITTI and Sintel, and a novel method for incorporating information from past frames into flow estimation is introduced.
PWOC-3D: Deep Occlusion-Aware End-to-End Scene Flow Estimation
TLDR
This paper proposes PWOC-3D, a compact CNN architecture to predict scene flow from stereo image sequences in an end-to-end supervised setting, and proposes a novel self-supervised strategy to predict occlusions from images (learned without any labeled occlusion data).
Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation
  • Junhwa Hur, S. Roth
  • Computer Science
    2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2019
TLDR
This work proposes an iterative residual refinement (IRR) scheme based on weight sharing that can be combined with several backbone networks and achieves state-of-the-art results for both optical flow and occlusion estimation across several standard datasets.
Occlusions, Motion and Depth Boundaries with a Generic Network for Disparity, Optical Flow or Scene Flow Estimation
TLDR
An efficient learning-based approach to estimate occlusion areas jointly with disparities or optical flow is presented and the estimated occlusions and motion boundaries clearly improve over the state-of-the-art.
SENSE: A Shared Encoder Network for Scene-Flow Estimation
TLDR
A compact network for holistic scene flow estimation is introduced, called SENSE, which shares common encoder features among four closely-related tasks: optical flow estimation, disparity estimation from stereo, occlusion estimation, and semantic segmentation, which leads to a compact and efficient model at inference time.
...
...

References

SHOWING 1-10 OF 55 REFERENCES
Occlusion-Aware Optical Flow Estimation
TLDR
This paper proposes a variational formulation that jointly computes optical flow, implicitly detects occlusions and extrapolates optical flow in occlusion areas, and shows significant improvements in the computed optical flow fields over other approaches, both qualitatively and quantitatively.
Determining Occlusions from Space and Time Image Reconstructions
TLDR
This work proposes a novel approach to occlusion detection where visibility or not of a point in next frame is formulated in terms of visual reconstruction, and outperforms state-of-the-art detection methods on the challenging MPI Sintel dataset.
Optical Flow with Geometric Occlusion Estimation and Fusion of Multiple Frames
TLDR
A framework for estimating optical flow which leads to improvements on these difficult cases by estimating occlusions and using additional temporal information, and a novel method of dealing with temporal information in image sequences by using “inertial estimates” of the flow.
Symmetrical Dense Optical Flow Estimation with Occlusions Detection
TLDR
A method to recover a dense optical flow field map from two images, while explicitely taking into account the symmetry across the images as well as possible occlusions in the flow field, and extended a classical approach to handle those.
Exploiting Semantic Information and Deep Matching for Optical Flow
TLDR
This work tackles the problem of estimating optical flow from a monocular camera in the context of autonomous driving by proposing to estimate the traffic participants using instance-level segmentation, and introduces a new convolutional net that learns to perform flow matching, and is able to estimates the uncertainty of its matches.
A TV-L1 Optical Flow Method with Occlusion Detection
TLDR
A variational model for joint optical flow and occlusion estimation based on the divergence of the flow and the proposed energy favors the location of occlusions on regions where this divergence is negative.
Optical Flow in Mostly Rigid Scenes
TLDR
An optical flow algorithm that estimates an explicit segmentation of moving objects from appearance and physical constraints and achieves state-of-the-art results on both the MPI-Sintel and KITTI-2015 benchmarks.
Efficient Joint Segmentation, Occlusion Labeling, Stereo and Flow Estimation
TLDR
A new optimization algorithm is proposed for the authors' SLIC-like objective which preserves connecteness of image segments and exploits shape regularization in the form of boundary length and can be achieved an order of magnitude faster than competing approaches.
Highly Overparameterized Optical Flow Using PatchMatch Belief Propagation
TLDR
It is shown that optimization over this high-dimensional, continuous state space can be carried out using an adaptation of the recently introduced PatchMatch Belief Propagation energy minimization algorithm, and that the resulting flow fields compare favorably to the state of the art on a number of small- and large-displacement datasets.
FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks
TLDR
The concept of end-to-end learning of optical flow is advanced and it work really well, and faster variants that allow optical flow computation at up to 140fps with accuracy matching the original FlowNet are presented.
...
...