Deep Online Fused Video Stabilization

@article{Shi2022DeepOF,
  title={Deep Online Fused Video Stabilization},
  author={Zhenmei Shi and Fuhao Shi and Wei-Sheng Lai and Chia-Kai Liang and Yingyu Liang},
  journal={2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year={2022},
  pages={865-873}
}
  • Zhenmei Shi, Fuhao Shi, Yingyu Liang
  • Published 2 February 2021
  • Computer Science
  • 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
We present a deep neural network (DNN) that uses both sensor data (gyroscope) and image content (optical flow) to stabilize videos through unsupervised learning. The network fuses optical flow with real/virtual camera pose histories into a joint motion representation. Next, the LSTM cell infers the new virtual camera pose, which is used to generate a warping grid that stabilizes the video frames. We adopt a relative motion representation as well as a multi-stage training strategy to optimize… 

Figures and Tables from this paper

References

SHOWING 1-10 OF 42 REFERENCES
Deep Online Video Stabilization With Multi-Grid Warping Transformation Learning
TLDR
This paper presents a video stabilization technique using a convolutional neural network that does not explicitly represent the camera path and does not use future frames, and is able to handle low-quality videos, such as night-scene videos, watermarked videos, blurry videos, and noisy videos.
Learning Video Stabilization Using Optical Flow
  • Ji-yang Yu, R. Ramamoorthi
  • Computer Science
    2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2020
TLDR
A novel neural network is proposed that infers the per-pixel warp fields for video stabilization from the optical flow fields of the input video, achieving quantitatively and visually better results than the state-of-the-art optimization based and deep learning based video stabilization methods.
PWStableNet: Learning Pixel-Wise Warping Maps for Video Stabilization
  • M. Zhao, Q. Ling
  • Computer Science
    IEEE Transactions on Image Processing
  • 2020
TLDR
This work proposes a novel video stabilization network, called PWStableNet, which comes up pixel-wise warping maps, i.e., potentially different warping for different pixels, and stabilizes each pixel to its stabilized view, and is believed to be the first deep learning based pixels-wise video stabilization.
Deep Video Stabilization Using Adversarial Networks
TLDR
A novel online deep learning framework to learn the stabilization transformation for each unsteady frame, given historical steady frames, composed of a generative network with spatial transformer networks embedded in different layers, and generates a stable frame for the incoming unstable frame by computing an appropriate affine transformation.
Robust Video Stabilization by Optimization in CNN Weight Space
  • Ji-yang Yu, R. Ramamoorthi
  • Computer Science
    2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2019
TLDR
This paper introduces a new formulation of the video stabilization task based on first principles, which leads to a large scale non-convex problem, and proposes a novel optimization routine that transfers this problem into the convolutional neural network parameter domain.
Deep Iterative Frame Interpolation for Full-frame Video Stabilization
TLDR
This work proposes an unsupervised deep approach to full-frame video stabilization that can generate video frames without cropping and low distortion, and utilizes frame interpolation techniques to generate in between frames, leading to reduced inter-frame jitter.
FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks
TLDR
The concept of end-to-end learning of optical flow is advanced and it work really well, and faster variants that allow optical flow computation at up to 140fps with accuracy matching the original FlowNet are presented.
5D Video Stabilization through Sensor Vision Fusion
TLDR
This paper model the residual camera motion after 3D rotation based stabilization as residual 2D translation in the image plane, which can be estimated using visual information.
Subspace video stabilization
TLDR
This article focuses on the problem of transforming a set of input 2D motion trajectories so that they are both smooth and resemble visually plausible views of the imaged scene, and offers the first method that both achieves high-quality video stabilization and is practical enough for consumer applications.
Video stabilization with a depth camera
TLDR
Though the depth image is noisy, incomplete and low resolution, it facilitates both camera motion estimation and frame warping, which make the video stabilization a much well posed problem.
...
...