AutoFlow: Learning a Better Training Set for Optical Flow

  title={AutoFlow: Learning a Better Training Set for Optical Flow},
  author={Deqing Sun and Daniel Vlasic and Charles Herrmann and V. Jampani and Michael Krainin and Huiwen Chang and Ramin Zabih and William T. Freeman and Ce Liu},
  journal={2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  • Deqing Sun, Daniel Vlasic, +6 authors Ce Liu
  • Published 29 April 2021
  • Computer Science
  • 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Synthetic datasets play a critical role in pre-training CNN models for optical flow, but they are painstaking to generate and hard to adapt to new applications. To automate the process, we present AutoFlow, a simple and effective method to render training data for optical flow that optimizes the performance of a model on a target dataset. AutoFlow takes a layered approach to render synthetic data, where the motion, shape, and appearance of each layer are controlled by learnable hyperparameters… Expand
Detail Preserving Residual Feature Pyramid Modules for Optical Flow
A novel Residual Feature Pyramid Module (RFPM) is proposed which retains important details in the feature map without changing the overall iterative refinement design of the optical flow estimation and is one of the top-performing methods in KITTI. Expand
Sensor-Guided Optical Flow
Given the availability of sparse yet accurate optical flow hints from an external source, these are injected to modulate the correlation scores computed by a state-of-the-art optical flow network and guide it towards more accurate predictions. Expand
Perceiver IO: A General Architecture for Structured Inputs & Outputs
The recently-proposed Perceiver model obtains good results on several domains while scaling linearly in compute and memory with the input size while learning to flexibly query the model’s latent space to produce outputs of arbitrary size and semantics by Perceiver IO. Expand
GMFlow: Learning Optical Flow via Global Matching
Learning-based optical flow estimation has been dominated with the pipeline of cost volume with convolutions for flow regression, which is inherently limited to local correlations and thus is hard toExpand


Learning to Generate 3D Training Data Through Hybrid Gradient
  • Dawei Yang, Jun Deng
  • Computer Science
  • 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2020
This work proposes a new method that optimizes the generation of 3D training data based on what it calls "hybrid gradient", which parametrize the design decisions as a real vector, and combines the approximate gradient and the analytical gradient to obtain the hybrid gradient of the network performance with respect to this vector. Expand
SelFlow: Self-Supervised Learning of Optical Flow
We present a self-supervised learning approach for optical flow. Our method distills reliable flow estimations from non-occluded pixels, and uses these predictions as ground truth to learn opticalExpand
RenderGAN: Generating Realistic Labeled Data
This work presents a novel framework called RenderGAN that can generate large amounts of realistic, labeled images by combining a 3D model and the Generative Adversarial Network framework, and applies it to generate images of barcode-like markers that are attached to honeybees. Expand
FlowNet: Learning Optical Flow with Convolutional Networks
This paper constructs CNNs which are capable of solving the optical flow estimation problem as a supervised learning task, and proposes and compares two architectures: a generic architecture and another one including a layer that correlates feature vectors at different image locations. Expand
AutoAugment: Learning Augmentation Policies from Data
This paper describes a simple procedure called AutoAugment to automatically search for improved data augmentation policies, which achieves state-of-the-art accuracy on CIFAR-10, CIFar-100, SVHN, and ImageNet (without additional data). Expand
LiteFlowNet: A Lightweight Convolutional Neural Network for Optical Flow Estimation
This paper presents an alternative network that attains performance on par with FlowNet2 on the challenging Sintel final pass and KITTI benchmarks, while being 30 times smaller in the model size and 1.36 times faster in the running speed. Expand
FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks
The concept of end-to-end learning of optical flow is advanced and it work really well, and faster variants that allow optical flow computation at up to 140fps with accuracy matching the original FlowNet are presented. Expand
What Makes Good Synthetic Training Data for Learning Disparity and Optical Flow Estimation?
This paper promotes the use of synthetically generated data for the purpose of training deep networks on visual recognition tasks and suggests multiple ways to generate such data and evaluates the influence of dataset properties on the performance and generalization properties of the resulting networks. Expand
Models Matter, So Does Training: An Empirical Study of CNNs for Optical Flow Estimation
A compact but effective CNN model, called PWC-Net, is designed according to simple and well-established principles: pyramidal processing, warping, and cost volume processing and is the winning entry in the optical flow competition of the robust vision challenge. Expand
Meta-Sim: Learning to Generate Synthetic Datasets
Meta-Sim is proposed, which learns a generative model of synthetic scenes, and obtain images as well as its corresponding ground-truth via a graphics engine, and can greatly improve content generation quality over a human-engineered probabilistic scene grammar. Expand