Hybrid Neural Fusion for Full-frame Video Stabilization

@article{Liu2021HybridNF,
  title={Hybrid Neural Fusion for Full-frame Video Stabilization},
  author={Yu-Lun Liu and Wei-Sheng Lai and Ming-Hsuan Yang and Yung-Yu Chuang and Jia-Bin Huang},
  journal={2021 IEEE/CVF International Conference on Computer Vision (ICCV)},
  year={2021},
  pages={2279-2288}
}
Existing video stabilization methods often generate visible distortion or require aggressive cropping of frame boundaries, resulting in smaller field of views. In this work, we present a frame synthesis algorithm to achieve full-frame video stabilization. We first estimate dense warp fields from neighboring frames and then synthesize the stabilized frame by fusing the warped contents. Our core technical novelty lies in the learning-based hybrid-space fusion that alleviates artifacts caused by… 
View on IEEE
View With Semantic Reader
6 Citations
Highly Influential Citations
1
Background Citations
4
Methods Citations
2

Figures and Tables from this paper

6 Citations

Self-Supervised Real-time Video Stabilization
TLDR
This work proposes a novel method of real-time video stabilization - transforming a shaky video to a stabilized video as if it were stabilized via gimbals inreal-time, which outperforms state-of-the-art real- time video stabilization methods as well as of course methods that require camera trajectory optimization.
Complete and temporally consistent video outpainting
TLDR
A novel method for video outpainting that is able to reconstruct the full frame, including unseen image parts, and is temporally consistent and able to successfully broaden the aspect ratio of a video.
Optical-Flow-Reuse-Based Bidirectional Recurrent Network for Space-Time Video Super-Resolution
TLDR
This paper proposes a coarseto-fine bidirectional recurrent neural network instead of using ConvLSTM to leverage knowledge between adjacent frames, which considerably reduces the computation burden of frame alignment compared with existing LSTM-based designs.
Video stabilization based on image registration
TLDR
The quantitative and qualitative evaluations show that the proposed registration-based video stabilization method can produce comparable results with the state-of-the-art methods and achieve more stable performance on challenging situations.
Boosting View Synthesis with Residual Transfer
TLDR
The core idea is to transfer color residuals (the difference between the input images and their reconstruction) from training views to novel views using a heuristic weighting scheme depending on ray visibility and angular differences.
Hierarchical Sampling based Particle Filter for Visual-inertial Gimbal in the Wild
TLDR
A CV based new tracking and fusion algorithm dedicated for gimbal system on drones operating in nature is proposed, main contributions are listed as below: a light-weight Resnet -18 backbone based network model was trained from scratch, and deployed onto Jetson Nano platform to segment the image into binary parts (ground and sky).

References

SHOWING 1-10 OF 88 REFERENCES
Deep Iterative Frame Interpolation for Full-frame Video Stabilization
TLDR
This work proposes an unsupervised deep approach to full-frame video stabilization that can generate video frames without cropping and low distortion, and utilizes frame interpolation techniques to generate in between frames, leading to reduced inter-frame jitter.
Learning Video Stabilization Using Optical Flow
  • Ji-yang Yu, R. Ramamoorthi
  • Computer Science
    2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2020
TLDR
A novel neural network is proposed that infers the per-pixel warp fields for video stabilization from the optical flow fields of the input video, achieving quantitatively and visually better results than the state-of-the-art optimization based and deep learning based video stabilization methods.
Full-frame video stabilization
TLDR
This work proposes a practical and robust approach of video stabilization that produces full-frame stabilized videos with good visual quality and develops a complete video stabilizer which can naturally keep the original image quality in the stabilized videos.
Deep Online Video Stabilization With Multi-Grid Warping Transformation Learning
TLDR
This paper presents a video stabilization technique using a convolutional neural network that does not explicitly represent the camera path and does not use future frames, and is able to handle low-quality videos, such as night-scene videos, watermarked videos, blurry videos, and noisy videos.
Video stabilization with a depth camera
TLDR
Though the depth image is noisy, incomplete and low resolution, it facilitates both camera motion estimation and frame warping, which make the video stabilization a much well posed problem.
Deep Video Stabilization Using Adversarial Networks
TLDR
A novel online deep learning framework to learn the stabilization transformation for each unsteady frame, given historical steady frames, composed of a generative network with spatial transformer networks embedded in different layers, and generates a stable frame for the incoming unstable frame by computing an appropriate affine transformation.
Temporally coherent completion of dynamic video
TLDR
An automatic video completion algorithm that synthesizes missing regions in videos in a temporally coherent fashion is presented that can handle dynamic scenes captured using a moving camera and jointly estimating optical flow and color in the missing regions.
Robust Video Stabilization by Optimization in CNN Weight Space
  • Ji-yang Yu, R. Ramamoorthi
  • Computer Science
    2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2019
TLDR
This paper introduces a new formulation of the video stabilization task based on first principles, which leads to a large scale non-convex problem, and proposes a novel optimization routine that transfers this problem into the convolutional neural network parameter domain.
Deep Video Deblurring for Hand-Held Cameras
TLDR
This work introduces a deep learning solution to video deblurring, where a CNN is trained end-to-end to learn how to accumulate information across frames, and shows that the features learned extend todeblurring motion blur that arises due to camera shake in a wide range of videos.
Plane-Based Content Preserving Warps for Video Stabilization
TLDR
A hybrid approach for novel view synthesis is presented, observing that the texture less regions often correspond to large planar surfaces in the scene, and how the segmentation information can be efficiently obtained and seamlessly integrated into the stabilization framework is demonstrated.
...
...