Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling

  title={Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling},
  author={Jingyun Liang and Andreas Lugmayr and K. Zhang and Martin Danelljan and Luc Van Gool and Radu Timofte},
  journal={2021 IEEE/CVF International Conference on Computer Vision (ICCV)},
Normalizing flows have recently demonstrated promising results for low-level vision tasks. For image super-resolution (SR), it learns to predict diverse photo-realistic high-resolution (HR) images from the low-resolution (LR) image rather than learning a deterministic mapping. For image rescaling, it achieves high accuracy by jointly modelling the downscaling and upscaling processes. While existing approaches employ specialized techniques for these two tasks, we set out to unify them in a… 

Figures and Tables from this paper

Rethinking the Adaptiveness of Conditional Flow A Novel Distribution Adaptation Strategy for Image Super-Resolution

The part of the latent variable that differs significantly between the latent variables of HR and the Bicubic of LR, contains mainly high-frequency information of the image, which can alleviate generation of confusing artifacts without reducing perceptual quality.

Enhancing Image Rescaling using Dual Latent Variables in Invertible Neural Network

Normalizing flow models have been used successfully for generative image super-resolution (SR) by approximating complex distribution of natural images to simple tractable distribution in latent space

Downscaled Representation Matters: Improving Image Rescaling with Collaborative Downscaled Images

A Hierarchical Collaborative Downscaling (HCD) method that performs gradient descent in both HR and LR domains to improve the downscaled representations and significantly improves the reconstruction performance both quantitatively and qualitatively.

Effective Invertible Arbitrary Image Rescaling

A simple and effective invertible arbitrary rescaling network (IARN) is proposed to achieve arbitrary image rescaling by training only one model in this work and is shown to achieve a state-of-the-art (SOTA) performance in bidirectional arbitrary rescalling without compromising perceptual quality in LR outputs.

LAR-SR: A Local Autoregressive Model for Image Super-Resolution

Experimental results show that the proposed LAR-SR approach achieves superior performance on the vi-sual quality and quantitative metrics compared with other generative models such as GAN, Flow, and is competitive with the mixture generative model.

Superresolution Reconstruction of Single Image for Latent features

The proposed Denoising Diffusion Probabilistic model for Latent features (LDDPM) reconstructs more realistic HR images and obtains better PSNR and SSIM performance compared to existing SISR tasks, thus providing a new idea for S ISR tasks.

Reference-based Image Super-Resolution with Deformable Attention Transformer

A deformable attention Transformer, namely DATSR, with multiple scales, each of which consists of a texture feature encoder (TFE) module, a reference-based deformable Attention (RDA) module and a residual feature aggregation (RFA) module is proposed.

Diverse Super-Resolution with Pretrained Hierarchical Variational Autoencoders

This work tackles the problem of diverse super-resolution by reusing VD-VAE, a state-of-the art variational autoencoder (VAE), finding that the hierarchical latent representation learned by VDVAE naturally separates the image low-frequency information from the image high-frequency details, deter-mined by the latent groups at the bottom of the latent hierarchy.

Normalizing Flow as a Flexible Fidelity Objective for Photo-Realistic Super-resolution

This work revisiting the L1 loss and showing that it corresponds to a one-layer conditional flow is shown to outperform state-of-the-art methods for photo-realistic super-resolution and demonstrate that the flexibility of deeper flows leads to better visual quality and consistency when combined with adversarial losses.

DSRGAN: Detail Prior-Assisted Perceptual Single Image Super-Resolution via Generative Adversarial Networks

This work proposes a novel model-based algorithm, named DSRGAN, which can extract the detail layer of an image efficiently and outperforms the state-of-the-art SISR methods on perceptual metrics, meanwhile achieving comparable results in terms of fidelity metrics.



SRFlow: Learning the Super-Resolution Space with Normalizing Flow

The proposed SRFlow is a normalizing flow based super-resolution method capable of learning the conditional distribution of the output given the low-resolution input, and directly accounts for the ill-posed nature of the problem, and learns to predict diverse photo-realistic high-resolution images.

Learning a Convolutional Neural Network for Image Compact-Resolution

The requirements of image CR are translated into operable optimization targets for training CNN-CR and the visual quality of the compact resolved image is ensured by constraining its difference from a naively downsampled version and the information loss of imageCR is measured by upsampling/super-resolving the compact-resolved image and comparing that to the original image.

Residual Dense Network for Image Super-Resolution

This paper proposes residual dense block (RDB) to extract abundant local features via dense connected convolutional layers and uses global feature fusion in RDB to jointly and adaptively learn global hierarchical features in a holistic way.

Task-Aware Image Downscaling

This paper proposes an auto-encoder-based framework that enables joint learning of the downscaling network and the upsc scaling network to maximize the restoration performance and validates the model’s generalization capability by applying it to the task of image colorization.

Learned Image Downscaling for Upscaling Using Content Adaptive Resampler

This paper proposes a learned image downscaling method based on content adaptive resampler (CAR) with consideration on the upscaling process and achieves a new state-of-the-art SR performance through upscaled guided image resamplers which adaptively preserve detailed information that is essential to theUpscaling.

Single image super-resolution from transformed self-exemplars

This paper expands the internal patch search space by allowing geometric variations, and proposes a compositional model to simultaneously handle both types of transformations to accommodate local shape variations.

Mutual Affine Network for Spatially Variant Kernel Estimation in Blind Image Super-Resolution

Extensive experiments show that the proposed MANet not only performs favorably for both spatially variant and invariant kernel estimation, but also leads to state-of-the-art blind SR performance when combined with non-blind SR methods.

Deep Unfolding Network for Image Super-Resolution

This paper proposes an end-to-end trainable unfolding network which leverages both learningbased methods and model-based methods to super-resolve blurry, noisy images for different scale factors via a single model, while maintaining the advantages of learning- based methods.

Learning a Deep Convolutional Network for Image Super-Resolution

This work proposes a deep learning method for single image super-resolution (SR) that directly learns an end-to-end mapping between the low/high-resolution images and shows that traditional sparse-coding-based SR methods can also be viewed as a deep convolutional network.

DeFlow: Learning Complex Image Degradations from Unpaired Data with Conditional Flows

DeFlow, a method for learning stochastic image degradations from unpaired data based on a novel unpaired learning formulation for conditional normalizing flows, is proposed, which allows to learn the conditional distribution of a noisy image given the clean input by solely minimizing the negative log-likelihood of the marginal distributions.