Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation

@article{Jiang2018SuperSH,
  title={Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation},
  author={Huaizu Jiang and Deqing Sun and V. Jampani and Ming-Hsuan Yang and Erik G. Learned-Miller and Jan Kautz},
  journal={2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2018},
  pages={9000-9008}
}
Given two consecutive frames, video interpolation aims at generating intermediate frame(s) to form both spatially and temporally coherent video sequences. [...] Key Method We start by computing bi-directional optical flow between the input images using a U-Net architecture. These flows are then linearly combined at each time step to approximate the intermediate bi-directional optical flows. These approximate flows, however, only work well in locally smooth regions and produce artifacts around motion boundaries…Expand
A Multi-frame Video Interpolation Neural Network for Large Motion
TLDR
This work combines multi-scale optical flow network PWC-Net and optimized network UNet++ to form a multi-frame interpolation neural network, which can be trained end-to-end and outperforms representative state-of-the-art methods, especially in large motion scenarios. Expand
Depth-Aware Video Frame Interpolation
TLDR
A video frame interpolation method which explicitly detects the occlusion by exploring the depth information, and develops a depth-aware flow projection layer to synthesize intermediate flows that preferably sample closer objects than farther ones. Expand
Adaptive Frame Interpolation using an End-to-End Deep Net with High Quality Flow Estimation
TLDR
An adaptive variable frame inter-polation method, which evaluates the amount of frame to be generated according to the estimated motion, to reduce the storage space and improve the generation efficiency and add edge loss to the loss function, expecting better results. Expand
Multi-Frame Pyramid Refinement Network for Video Frame Interpolation
TLDR
Experimental results on several independent datasets show that the proposed multi-frame pyramid refinement network can effectively handle challenging cases, and perform consistently better than other state-of-the-art methods. Expand
Deep Video Frame Interpolation Using Cyclic Frame Generation
TLDR
A new loss term, the cycle consistency loss, which can better utilize the training data to not only enhance the interpolation results, but also maintain the performance better with less training data is introduced. Expand
Flow-aware synthesis: A generic motion model for video frame interpolation
TLDR
Qualitative and quantitative experimental results show that the generic adaptive flow prediction module presented can produce high-quality results and outperforms the existing state-of-the-art methods on popular public datasets. Expand
Fine-Grained Motion Estimation for Video Frame Interpolation
TLDR
This article proposes a novel fine-grained motion estimation approach (FGME) for video frame interpolation that mainly contains two strategies: multi-scale coarse-to-fine optimization and multiple motion features estimation. Expand
Video Frame Interpolation via Cyclic Fine-Tuning and Asymmetric Reverse Flow
TLDR
This work uses a convolutional neural network that takes two frames as input and predicts two optical flows with pixelwise weights that outperforms the publicly available state-of-the-art methods on multiple datasets. Expand
Multiframe interpolation for video using phase features
TLDR
A phase-feature-aided multiframe interpolation network that aims to estimate multiple in-between frames in one pass and handle challenging scenarios such as extreme light changes and occlusion and achieves marked visual improvement in challenging scenarios is presented. Expand
PoSNet: 4x Video Frame Interpolation Using Position-Specific Flow
TLDR
A 4× video-interpolation framework that aims to convert 15-fps to 60-fps videos based on a structure comprising flow estimation followed by an enhancement network that interpolates high-quality frames with a fast runtime and demonstrates its superiority in the AIM 2019 video temporal super-resolution challenge. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 41 REFERENCES
Video Frame Interpolation via Adaptive Convolution
TLDR
This paper presents a robust video frame interpolation method that considers pixel synthesis for the interpolated frame as local convolution over two input frames and employs a deep fully convolutional neural network to estimate a spatially-adaptive convolution kernel for each pixel. Expand
Video Frame Interpolation via Adaptive Separable Convolution
TLDR
This paper develops a deep fully convolutional neural network that takes two input frames and estimates pairs of 1D kernels for all pixels simultaneously, which allows for the incorporation of perceptual loss to train the neural network to produce visually pleasing frames. Expand
Video Frame Synthesis Using Deep Voxel Flow
TLDR
This work addresses the problem of synthesizing new video frames in an existing video, either in-between existing frames (interpolation), or subsequent to them (extrapolation), by training a deep network that learns to synthesize video frames by flowing pixel values from existing ones, which is called deep voxel flow. Expand
Phase-based frame interpolation for video
TLDR
A novel, bounded phase shift correction method that combines phase information across the levels of a multi-scale pyramid is introduced that allows in-between images to be generated by simple per-pixel phase modification, without the need for any form of explicit correspondence estimation. Expand
Spatio-temporal video autoencoder with differentiable memory
TLDR
One direct application of the proposed framework in weakly-supervised semantic segmentation of videos through label propagation using optical flow is presented, using as temporal decoder a robust optical flow prediction module together with an image sampler serving as built-in feedback loop. Expand
Deep Video Deblurring
TLDR
This work introduces a deep learning solution to video deblurring, where a CNN is trained end-to-end to learn how to accumulate information across frames, and shows that the features learned extend todeblurring motion blur that arises due to camera shake in a wide range of videos. Expand
Moving gradients: a path-based method for plausible image interpolation
TLDR
A method for plausible interpolation of images, with a wide range of applications like temporal up-sampling for smooth playback of lower frame rate video, smooth view interpolation, and animation of still images, to allow arbitrary (asymmetric) transition points, where the path moves from one image to the other. Expand
Slow Flow: Exploiting High-Speed Cameras for Accurate and Diverse Optical Flow Reference Data
TLDR
This paper tracks pixels through densely sampled space-time volumes recorded with a high-speed video camera to exploit the linearity of small motions and reasons about occlusions from multiple frames, and establishes accurate reference flow fields outside the laboratory in natural environments. Expand
Dual Motion GAN for Future-Flow Embedded Video Prediction
TLDR
A dual motion Generative Adversarial Net architecture is developed, which learns to explicitly enforce future-frame predictions to be consistent with the pixel-wise flows in the video through a duallearning mechanism. Expand
Optical Flow Estimation Using a Spatial Pyramid Network
TLDR
The Spatial Pyramid Network (SPyNet) is much simpler and 96% smaller than FlowNet in terms of model parameters, which makes it more efficient and appropriate for embedded applications. Expand
...
1
2
3
4
5
...