Unsupervised Video Interpolation Using Cycle Consistency

@article{Reda2019UnsupervisedVI,
  title={Unsupervised Video Interpolation Using Cycle Consistency},
  author={Fitsum A. Reda and Deqing Sun and Aysegul Dundar and Mohammad Shoeybi and Guilin Liu and Kevin J. Shih and Andrew Tao and Jan Kautz and Bryan Catanzaro},
  journal={2019 IEEE/CVF International Conference on Computer Vision (ICCV)},
  year={2019},
  pages={892-900}
}
Learning to synthesize high frame rate videos via interpolation requires large quantities of high frame rate training videos, which, however, are scarce, especially at high resolutions. [...] Key Method This simple unsupervised constraint alone achieves results comparable with supervision using the ground truth intermediate frames. We further introduce a pseudo supervised loss term that enforces the interpolated frames to be consistent with predictions of a pre-trained interpolation model.Expand
Video Frame Interpolation and Enhancement via Pyramid Recurrent Framework
TLDR
This paper develops a frame interpolation module with a pyramid structure to cyclically synthesize high-quality intermediate frames and proposes an inter-pyramid recurrent module to connect sequential models to exploit the temporal relationship.
Training Weakly Supervised Video Frame Interpolation with Events
  • Zhiyang Yu, Yu Zhang, +4 authors Jimmy Ren
  • 2021
Event-based video frame interpolation is promising as event cameras capture dense motion signals that can greatly facilitate motion-aware synthesis. However, training existing frameworks for this
Blurry Video Frame Interpolation
TLDR
A pyramid module to cyclically synthesize clear intermediate frames to reduce motion blur and up-convert frame rate simultaneously is developed and an inter-pyramid recurrent module is proposed to connect sequential models to exploit the temporal relationship.
Scene-Adaptive Video Frame Interpolation via Meta-Learning
TLDR
This work shows the benefits of 'test-time adaptation' through simple fine-tuning of a network, then improves its efficiency by incorporating meta-learning, and obtains significant performance gains with only a single gradient update without any additional parameters.
Context-based video frame interpolation via depthwise over-parameterized convolution
TLDR
Experimental results demonstrate that the proposed context-based video frame interpolation method via depthwise over-parameterized convolution performs qualitatively and quantitatively better than state-of-the-art methods.
Spatio-Temporal Multi-Flow Network for Video Frame Interpolation
TLDR
This work presents a novel deep learning based VFI method, ST-MFNet, based on a Spatio-Temporal Multi-Flow architecture, which employs a new multi-scale multi-flow predictor to estimate many-to-one intermediate flows, and a 3D CNN is employed to model the content dynamics over an extended temporal window.
A
  • Q. N. Tran, Shih-Hsuan Yang
  • Shape Classification and Analysis
  • 2018
Frame interpolation, which generates an intermediate frame given adjacent ones, finds various applications such as frame rate up-conversion, video compression, and video streaming. Instead of using
Asymmetric Bilateral Motion Estimation for Video Frame Interpolation
TLDR
A novel video frame interpolation algorithm based on asymmetric bilateral motion estimation (ABME), which synthesizes an intermediate frame between two input frames, is proposed, which achieves excellent performance on various datasets.
Video Interpolation and Prediction with Unsupervised Landmarks
TLDR
This work poses video prediction and interpolation as unsupervised latent structure inference followed by a temporal prediction in this latent space, and is able to achieve high-quality long-range video interpolation and extrapolation by operating on the landmark representation space.
Enhanced Real-Time Intermediate Flow Estimation for Video Frame Interpolation
  • Minseop Kim, Haechul Choi
  • Computer Science
    J. Web Eng.
  • 2021
TLDR
A bidirectional intermediate flow estimation method for real-time video frame interpolation is introduced and shows a high prediction accuracy and the interpolation speed of 84 fps was achieved for 480p resolution.
...
1
2
3
4
...

References

SHOWING 1-10 OF 28 REFERENCES
Deep Video Frame Interpolation Using Cyclic Frame Generation
TLDR
A new loss term, the cycle consistency loss, which can better utilize the training data to not only enhance the interpolation results, but also maintain the performance better with less training data is introduced.
Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation
TLDR
This work proposes an end-to-end convolutional neural network for variable-length multi-frame video interpolation, where the motion interpretation and occlusion reasoning are jointly modeled.
Video Frame Interpolation via Adaptive Convolution
TLDR
This paper presents a robust video frame interpolation method that considers pixel synthesis for the interpolated frame as local convolution over two input frames and employs a deep fully convolutional neural network to estimate a spatially-adaptive convolution kernel for each pixel.
Video Frame Synthesis Using Deep Voxel Flow
TLDR
This work addresses the problem of synthesizing new video frames in an existing video, either in-between existing frames (interpolation), or subsequent to them (extrapolation), by training a deep network that learns to synthesize video frames by flowing pixel values from existing ones, which is called deep voxel flow.
Video Frame Interpolation via Adaptive Separable Convolution
TLDR
This paper develops a deep fully convolutional neural network that takes two input frames and estimates pairs of 1D kernels for all pixels simultaneously, which allows for the incorporation of perceptual loss to train the neural network to produce visually pleasing frames.
Context-Aware Synthesis for Video Frame Interpolation
  • Simon Niklaus, Feng Liu
  • Computer Science
    2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
  • 2018
TLDR
A context-aware synthesis approach that warps not only the input frames but also their pixel-wise contextual information and uses them to interpolate a high-quality intermediate frame and outperforms representative state-of-the-art approaches.
SDC-Net: Video Prediction Using Spatially-Displaced Convolution
TLDR
SDC module for video frame prediction with spatially-displaced convolution inherits the merits of both vector-based and kernel-based approaches, while ameliorating their respective disadvantages.
Deep Video Deblurring for Hand-Held Cameras
TLDR
This work introduces a deep learning solution to video deblurring, where a CNN is trained end-to-end to learn how to accumulate information across frames, and shows that the features learned extend todeblurring motion blur that arises due to camera shake in a wide range of videos.
Learning Image Matching by Simply Watching Video
TLDR
An unsupervised learning based approach to the ubiquitous computer vision problem of image matching that achieves surprising performance comparable to traditional empirically designed methods.
High-quality video view interpolation using a layered representation
TLDR
This paper shows how high-quality video-based rendering of dynamic scenes can be accomplished using multiple synchronized video streams combined with novel image-based modeling and rendering algorithms, and develops a novel temporal two-layer compressed representation that handles matting.
...
1
2
3
...