GMFlow: Learning Optical Flow via Global Matching

@article{Xu2021GMFlowLO,
  title={GMFlow: Learning Optical Flow via Global Matching},
  author={Haofei Xu and Jing Zhang and Jianfei Cai and Hamid Rezatofighi and Dacheng Tao},
  journal={2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2021},
  pages={8111-8120}
}
  • Haofei XuJing Zhang D. Tao
  • Published 26 November 2021
  • Computer Science
  • 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Learning-based optical flow estimation has been dominated with the pipeline of cost volume with convolutions for flow regression, which is inherently limited to local correlations and thus is hard to address the long-standing challenge of large displacements. To alleviate this, the state-of-the-art framework RAFT gradually improves its prediction quality by using a large number of iterative refinements, achieving remarkable performance but introducing linearly increasing inference time. To… 
[PDF] Semantic Reader
53 Citations
Highly Influential Citations
7
Background Citations
27
Methods Citations
28

Figures and Tables from this paper

53 Citations

Rethinking Optical Flow from Geometric Matching Consistent Perspective

The proposed MatchFlow model employs a QuadTree attention-based network pre-trained on MegaDepth to extract coarse features for further flow regression, and has great cross-dataset generalization.

PatchFlow: A Two-Stage Patch-Based Approach for Lightweight Optical Flow Estimation

This work redistributes the regular CNN-based optical flow regression into a two-stage pipeline, where the first stage estimates an optical flow for a low-resolution image version, and shows that such a pipeline can additionally allow for utilizing a lighter and shallower model in the two stages.

GMSF: Global Matching Scene Flow

This work proposes to decompose the feature extraction step via a hybrid local-global-cross transformer architecture which is crucial to accurate and robust feature representations and sets a new state-of-the-art on multiple scene flow estimation benchmarks.

AnyFlow: Arbitrary Scale Optical Flow with Implicit Neural Representation

AnyFlow is introduced, a robust network that estimates accurate flow from images of various resolutions that establishes a new state-of-the-art performance of cross-dataset generalization on the KITTI dataset, while achieving comparable accuracy on the online benchmarks to other SOTA methods.

Optical Flow for Autonomous Driving: Applications, Challenges and Improvements

This work proposes and evaluates training strategies to improve a learning-based optical flow algorithm by leveraging the only existing fisheye dataset with optical flow ground truth, and proposes a novel, generic semi-supervised framework that significantly boosts performances of existing methods in such conditions.

CGCV: Context Guided Correlation Volume for Optical Flow Neural Networks

This paper investigates the functionality of RAFT’s context encoder, then proposes a new Context Guided Correlation Volume (CGCV), which can be universally integrated with RAFT-based computation methods for enhanced performance, especially effective in the presence of motion blur, de-focus blur and atmospheric effects.

DCVNet: Dilated Cost Volume Networks for Fast Optical Flow

This paper proposes an alternative to coarse-to-fine or recurrent processing of the cost volume by constructing cost volumes with different dilation factors to capture small and large displacements simultaneously, which obtains comparable accuracy to existing approaches and achieves real-time inference.

Towards Equivariant Optical Flow Estimation with Deep Learning

An extensive evaluation of the sign imbalance for the state-of-the-art optical flow estimators is provided and two strategies to mitigate the phenomenon are proposed, i) by constraining the model estimations during inference, and, ii) byconstraining the loss function during training.

FlowFormer++: Masked Cost Volume Autoencoding for Pretraining Optical Flow Estimation

A novel pre-text reconstruction task, which encourages the cost-volume encoder to aggregate long-range information and ensures pretraining-finetuning consistency, is proposed and shown how to modify the FlowFormer architecture to accommodate masks during pretraining.

FlowNAS: Neural Architecture Search for Optical Flow Estimation

A neural architecture search method named FlowNAS to automatically find the better encoder architecture for flow estimation task, and proposes Feature Alignment Distillation, which utilizes a well-trained flow estimator to guide the training of super-network.
...

54 References

Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation

    Junhwa HurS. Roth
    Computer Science
    2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2019
This work proposes an iterative residual refinement (IRR) scheme based on weight sharing that can be combined with several backbone networks and achieves state-of-the-art results for both optical flow and occlusion estimation across several standard datasets.

LiteFlowNet3: Resolving Correspondence Ambiguity for More Accurate Optical Flow Estimation

LiteFlowNet3, a deep network consisting of two specialized modules, is introduced, to address the issue of outliers in the cost volume by amending each cost vector through an adaptive modulation prior to the flow decoding and improve the flow accuracy by exploring local flow consistency.

FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks

The concept of end-to-end learning of optical flow is advanced and it work really well, and faster variants that allow optical flow computation at up to 140fps with accuracy matching the original FlowNet are presented.

DeepFlow: Large Displacement Optical Flow with Deep Matching

This work proposes a descriptor matching algorithm, tailored to the optical flow problem, that allows to boost performance on fast motions, and sets a new state-of-the-art on the MPI-Sintel dataset.

Displacement-Invariant Matching Cost Learning for Accurate Optical Flow Estimation

This paper proposes a novel solution that is able to bypass the requirement of building a 5D feature volume while still allowing the network to learn suitable matching costs from data, and achieves state-of-the-art accuracy on various datasets.

High-Resolution Optical Flow from 1D Attention and Correlation

A novel 1D formulation empowers the proposed method to scale to very high-resolution input images while maintaining competitive performance, and to achieve 2D correspondence modeling effect.

UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss

This work designs an unsupervised loss based on occlusion-aware bidirectional flow estimation and the robust census transform to circumvent the need for ground truth flow, enabling generic pre-training of supervised networks for datasets with limited amounts of ground truth.

Volumetric Correspondence Networks for Optical Flow

Several simple modifications that dramatically simplify the use of volumetric layers are introduced that significantly improve accuracy over SOTA on standard benchmarks while being significantly easier to work with - training converges in 10X fewer iterations, and most importantly, the networks generalize across correspondence tasks.

Learning to Estimate Hidden Motions with Global Motion Aggregation

This paper argues that the occlusion problem can be better solved in the two-frame case by modelling image self-similarities, and introduces a global motion aggregation module, a transformer-based approach to find long-range dependencies between pixels in the first image, and perform global aggregation on the corresponding motion features.

Models Matter, So Does Training: An Empirical Study of CNNs for Optical Flow Estimation

A compact but effective CNN model, called PWC-Net, is designed according to simple and well-established principles: pyramidal processing, warping, and cost volume processing and is the winning entry in the optical flow competition of the robust vision challenge.
...