Video Super Resolution Based on Deep Learning: A comprehensive survey

@article{Liu2022VideoSR,
  title={Video Super Resolution Based on Deep Learning: A comprehensive survey},
  author={Hongying Liu and Zhubo Ruan and Peng Zhao and Fanhua Shang and Linlin Yang and Yuanyuan Liu},
  journal={ArXiv},
  year={2022},
  volume={abs/2007.12928}
}
In recent years, deep learning has made great progress in the fields of image recognition, video analysis, natural language processing and speech recognition, including video super-resolution tasks. In this survey, we comprehensively investigate 28 state-of-the-art video super-resolution methods based on deep learning. It is well known that the leverage of information within video frames is important for video super-resolution. Hence we propose a taxonomy and classify the methods into six sub… 
Large Motion Video Super-Resolution with Dual Subnet and Multi-Stage Communicated Upsampling
TLDR
A novel deep neural network with Dual Subnet and Multi-stage Communicated Upsampling (DSMC) for super-resolution of videos with large motion and a novel dual subnet is devised to aid the training of the DSMC, whose dual loss helps to reduce the solution space as well as enhance the generalization ability.
Deep Learning-based Face Super-resolution: A Survey
TLDR
This survey presents a comprehensive review of deep learning-based FSR methods in a systematic manner and roughly categorizes existing methods according to the utilization of facial characteristics.
A Single Frame and Multi-Frame Joint Network for 360-degree Panorama Video Super-Resolution
TLDR
This is the first attempt to settle the super-resolution of spherical videos, and a novel single frame and multi-frame joint network (SMFN) for recovering high-resolution spherical videos from low-resolution inputs is proposed.
Finite-Exemplar-Based Super-Resolution for Specific Gaming Videos
TLDR
A high-frequency matching and fusion module is proposed to reduce the computational cost of patch-by-patch matching process and a dual-branch reconstruction module is designed to promote the exchange of information between low-resolution and reference features.
Rethinking Alignment in Video Super-Resolution Transformers
TLDR
This paper rethink the role of alignment in VSR Transformers and proposes a new andcient alignment method called patch alignment, which aligns image patches instead of pixels, which could demonstrate state-of-the-art performance on multiple benchmarks.
ADAPTIVE IMAGE SUPER-RESOLUTION ALGORITHM BASED ON FRACTIONAL FOURIER TRANSFORM
Super-resolution imaging is a critical image processing stage that improves visual image quality. Super-resolution imaging has a wide array of use in different fields, such as medical, satellite, and
Continuous Sign Language Recognition via Temporal Super-Resolution Network
TLDR
A temporal super-resolution network (TSRNet) is proposed, which takes the error rate between the estimated word error rate (WER) and the reference WER obtained by the reconstructed frame-level feature sequence and the complete original frame- level feature sequence as the WERD.
Recurrent Video Restoration Transformer with Guided Deformable Attention
TLDR
Extensive experiments on video super-resolution, deblurring, and denoising show that the proposed RVRT achieves state-of-the-art performance on benchmark datasets with balanced model size, testing memory and runtime.
Accelerating the Training of Video Super-Resolution Models
TLDR
This work shows that it is possible to gradually train video models from small to large spatial/temporal sizes, i.e .
...
...

References

SHOWING 1-10 OF 173 REFERENCES
Deep Learning for Image Super-Resolution: A Survey
TLDR
A survey on recent advances of image super-resolution techniques using deep learning approaches in a systematic way, which can roughly group the existing studies of SR techniques into three major categories: supervised SR, unsupervised SR, and domain-specific SR.
Enhanced Deep Residual Networks for Single Image Super-Resolution
TLDR
This paper develops an enhanced deep super-resolution network (EDSR) with performance exceeding those of current state-of-the-art SR methods, and proposes a new multi-scale deepsuper-resolution system (MDSR) and training method, which can reconstruct high-resolution images of different upscaling factors in a single model.
Video Super-Resolution With Convolutional Neural Networks
TLDR
This paper proposes a CNN that is trained on both the spatial and the temporal dimensions of videos to enhance their spatial resolution and shows that by using images to pretrain the model, a relatively small video database is sufficient for the training of the model to achieve and improve upon the current state-of-the-art.
Fast Spatio-Temporal Residual Network for Video Super-Resolution
TLDR
A novel fast spatio-temporal residual network (FSTRN) to adopt 3D convolutions for the video SR task in order to enhance the performance while maintaining a low computational load and significantly outperforms the current state-of-the-art methods.
Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network
TLDR
This paper presents the first convolutional neural network capable of real-time SR of 1080p videos on a single K2 GPU and introduces an efficient sub-pixel convolution layer which learns an array of upscaling filters to upscale the final LR feature maps into the HR output.
Deep Video Super-Resolution Network Using Dynamic Upsampling Filters Without Explicit Motion Compensation
TLDR
A novel end-to-end deep neural network that generates dynamic upsampling filters and a residual image, which are computed depending on the local spatio-temporal neighborhood of each pixel to avoid explicit motion compensation is proposed.
Video Super-Resolution via Deep Draft-Ensemble Learning
TLDR
This work proposes a new direction for fast video super-resolution via a SR draft ensemble, which is defined as the set of high-resolution patch candidates before final image deconvolution, and combines SR drafts through the nonlinear process in a deep convolutional neural network (CNN).
Video Super-Resolution Based on 3D-CNNS with Consideration of Scene Change
TLDR
This work proposes an effective 3D-CNN for video super-resolution that does not require motion alignment as preprocessing, maintains the temporal depth of spatio-temporal feature maps to maximally capture the temporally nonlinear characteristics between low and high resolution frames, and adopts residual learning in conjunction with the sub-pixel outputs.
Real-time video super-resolution via motion convolution kernel estimation
Deep Learning for Single Image Super-Resolution: A Brief Review
TLDR
This survey reviews representative deep learning-based SISR methods and group them into two categories according to their contributions to two essential aspects of S ISR: The exploration of efficient neural network architectures for SISS and the development of effective optimization objectives for deep SISr learning.
...
...