Depth-Aware Video Frame Interpolation

@article{Bao2019DepthAwareVF,
  title={Depth-Aware Video Frame Interpolation},
  author={Wenbo Bao and Wei-Sheng Lai and Chao Ma and Xiaoyun Zhang and Zhiyong Gao and Ming-Hsuan Yang},
  journal={2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2019},
  pages={3698-3707}
}
Video frame interpolation aims to synthesize nonexistent frames in-between the original frames. [...] Key Method In addition, we learn hierarchical features to gather contextual information from neighboring pixels. The proposed model then warps the input frames, depth maps, and contextual features based on the optical flow and local interpolation kernels for synthesizing the output frame. Our model is compact, efficient, and fully differentiable. Quantitative and qualitative results demonstrate that the…Expand
FeatureFlow: Robust Video Interpolation via Structure-to-Texture Generation
TLDR
This work devised a novel structure-to-texture generation framework which splits the video interpolation task into two stages: structure-guided interpolation and texture refinement, and is the first work that attempts to directly generate the intermediate frame through blending deep features. Expand
Blurry Video Frame Interpolation
TLDR
A pyramid module to cyclically synthesize clear intermediate frames to reduce motion blur and up-convert frame rate simultaneously is developed and an inter-pyramid recurrent module is proposed to connect sequential models to exploit the temporal relationship. Expand
Fine-Grained Motion Estimation for Video Frame Interpolation
TLDR
This article proposes a novel fine-grained motion estimation approach (FGME) for video frame interpolation that mainly contains two strategies: multi-scale coarse-to-fine optimization and multiple motion features estimation. Expand
Residual Learning of Video Frame Interpolation Using Convolutional LSTM
TLDR
This paper proposes a method that directly generates the intermediate frame from two consecutive frames and takes the average of these two frames and utilizes residual learning to learn the difference between the average and the ground truth middle frame. Expand
PoSNet: 4x Video Frame Interpolation Using Position-Specific Flow
TLDR
A 4× video-interpolation framework that aims to convert 15-fps to 60-fps videos based on a structure comprising flow estimation followed by an enhancement network that interpolates high-quality frames with a fast runtime and demonstrates its superiority in the AIM 2019 video temporal super-resolution challenge. Expand
Channel Attention Is All You Need for Video Frame Interpolation
TLDR
A simple but effective deep neural network for video frame interpolation, which is end-to-end trainable and is free from a motion estimation network component, and achieves outstanding performance compared to the existing models with a component for optical flow computation. Expand
Video Frame Interpolation Via Residue Refinement
  • Haopeng Li, Yuan Yuan, Qi Wang
  • Computer Science
  • ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
  • 2020
TLDR
A novel network structure that leverages residue refinement and adaptive weight to synthesize in-between frames and the adaptive weight map combines the forward and backward warped frames to reduce the artifacts is proposed. Expand
An Analytical Study of CNN-based Video Frame Interpolation Techniques
TLDR
How some deep convolution networks based methods have evolved over the years to improve the quality of the synthesized frames both qualitatively and quantitatively for video frame interpolation task is discussed. Expand
High-quality Frame Interpolation via Tridirectional Inference
TLDR
This work proposes a frame interpolation method by utilizing tridirectional information obtained from three input frames to learn rich and reliable inter-frame motion representations, including subtle nonlinear movement, which can be easily trained via any video frames in a self-supervised manner. Expand
Residual Learning of Video Frame Interpolation Using Convolutional LSTM
TLDR
The proposed method without explicit motion estimation can perform favorably against other state-of-the-art frame interpolation methods and utilizes residual learning to learn the difference between the average of these frames and the ground truth middle frame. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 43 REFERENCES
Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation
TLDR
This work proposes an end-to-end convolutional neural network for variable-length multi-frame video interpolation, where the motion interpretation and occlusion reasoning are jointly modeled. Expand
Video Frame Synthesis Using Deep Voxel Flow
TLDR
This work addresses the problem of synthesizing new video frames in an existing video, either in-between existing frames (interpolation), or subsequent to them (extrapolation), by training a deep network that learns to synthesize video frames by flowing pixel values from existing ones, which is called deep voxel flow. Expand
Context-Aware Synthesis for Video Frame Interpolation
  • Simon Niklaus, Feng Liu
  • Computer Science
  • 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
  • 2018
TLDR
A context-aware synthesis approach that warps not only the input frames but also their pixel-wise contextual information and uses them to interpolate a high-quality intermediate frame and outperforms representative state-of-the-art approaches. Expand
Video Frame Interpolation via Adaptive Convolution
TLDR
This paper presents a robust video frame interpolation method that considers pixel synthesis for the interpolated frame as local convolution over two input frames and employs a deep fully convolutional neural network to estimate a spatially-adaptive convolution kernel for each pixel. Expand
Video Frame Interpolation via Adaptive Separable Convolution
TLDR
This paper develops a deep fully convolutional neural network that takes two input frames and estimates pairs of 1D kernels for all pixels simultaneously, which allows for the incorporation of perceptual loss to train the neural network to produce visually pleasing frames. Expand
MEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Interpolation and Enhancement
TLDR
A novel adaptive warping layer is developed to integrate both optical flow and interpolation kernels to synthesize target frame pixels and is fully differentiable such that both the flow and kernel estimation networks can be optimized jointly. Expand
Video Enhancement with Task-Oriented Flow
TLDR
T task-oriented flow (TOFlow), a motion representation learned in a self-supervised, task-specific manner, is proposed, which outperforms traditional optical flow on standard benchmarks as well as the Vimeo-90K dataset in three video processing tasks. Expand
Depth Transfer: Depth Extraction from Video Using Non-Parametric Sampling
TLDR
The technique can be used to automatically convert a monoscopic video into stereo for 3D visualization, and is demonstrated through a variety of visually pleasing results for indoor and outdoor scenes, including results from the feature film Charade. Expand
Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields
TLDR
A deep convolutional neural field model for estimating depths from single monocular images, aiming to jointly explore the capacity of deep CNN and continuous CRF is presented, and a deep structured learning scheme which learns the unary and pairwise potentials of continuousCRF in a unified deep CNN framework is proposed. Expand
Unsupervised Learning of Depth and Ego-Motion from Video
TLDR
Empirical evaluation demonstrates the effectiveness of the unsupervised learning framework for monocular depth performs comparably with supervised methods that use either ground-truth pose or depth for training, and pose estimation performs favorably compared to established SLAM systems under comparable input settings. Expand
...
1
2
3
4
5
...