Model-Based Image Signal Processors via Learnable Dictionaries

  title={Model-Based Image Signal Processors via Learnable Dictionaries},
  author={Marcos V. Conde and Steven G. McDonagh and Matteo Maggioni and Alevs Leonardis and Eduardo P'erez-Pellitero},
Digital cameras transform sensor RAW readings into RGB images by means of their Image Signal Processor (ISP). Computational photography tasks such as image denoising and colour constancy are commonly performed in the RAW domain, in part due to the inherent hardware design, but also due to the appealing simplicity of noise statistics that result from the direct sensor readings. Despite this, the availability of RAW images is limited in comparison with the abundance and diversity of available RGB… 

Figures and Tables from this paper

Reversed Image Signal Processing and RAW Reconstruction. AIM 2022 Challenge Report

The proposed methods and benchmark establish the state-of-the-art for this low-level vision inverse problem, and generating realistic raw sensor readings can potentially benefit other tasks such as denoising and super-resolution.

Efficient Visual Computing with Camera RAW Snapshots

An unpaired CycleR2R network based on unsupervised CycleGAN is developed to train modular unrolled ISP and inverse ISP models using unpaired RAW and RGB images and Quantitative results reveal that RAW-domain task inference provides better detection accuracy and compression compared to RGB-domain processing.

Neural Photo-Finishing

Image processing pipelines are ubiquitous and we rely on them either directly, by filtering or adjusting an image post-capture, or indirectly, as image signal processing (ISP) pipelines on broadly

Reversing Image Signal Processors by Reverse Style Transferring

This work proposes a novel architecture, namely RST-ISP-Net, for learning to reverse the ISP operations with the help of adaptive feature normalization, and formulates this problem as a reverse style transferring and follows the practice used in the prior work.

Overexposure Mask Fusion: Generalizable Reverse ISP Multi-Step Refinement

The proposed methodology is a state-of-the-art solution to the task of RAW reconstruction, and the multi-step refinement process integrating an overexposure mask is novel in three ways: the pipeline is a generalizable process of refinement that can enhance other high performance methodologies that support end-to-end learning.

Swin2SR: SwinV2 Transformer for Compressed Image Super-Resolution and Restoration

The novel Swin Transformer V2 is explored, to improve SwinIR for image super-resolution, and in particular, the compressed input scenario, and the Swin2SR method, which is a top-5 solution at the AIM 2022 Challenge on Super-Resolution of Compressed Image and Video.

MIPI 2022 Challenge on Under-Display Camera Image Restoration: Methods and Results

The first MIPI challenge including five tracks focusing on novel image sensors and imaging algorithms and the developed solutions achieved state-of-the-art performance on Under-Display Camera Image Restoration is introduced.

Perceptual Image Enhancement for Smartphone Real-Time Applications

This paper proposes LPIENet, a lightweight network for perceptual image enhancement, with the focus on deploying it on smartphones, and shows that, with much fewer parameters and operations, the model can deal with the mentioned artifacts and achieve competitive performance compared with state-of-the-art methods on standard benchmarks.

Learned Smartphone ISP on Mobile GPUs with Deep Learning, Mobile AI & AIM 2022 Challenge: Report

The target was to develop an efficient end-to-end AI-based image signal processing (ISP) pipeline replacing the standard mobile ISPs that can run on modern smartphone GPUs using TensorFlow Lite.

Real-Time Under-Display Cameras Image Restoration and HDR on Mobile Devices

This work proposes a lightweight model for blind UDC Image Restoration and HDR, and provides a benchmark comparing the performance and runtime of different methods on smartphones.



Unprocessing Images for Learned Raw Denoising

This work presents a technique to “unprocess” images by inverting each step of an image processing pipeline, thereby allowing us to synthesize realistic raw sensor measurements from commonly available Internet photos.

Deep joint demosaicking and denoising

A new data-driven approach forDemosaicking and denoising is introduced: a deep neural network is trained on a large corpus of images instead of using hand-tuned filters and this network and training procedure outperform state-of-the-art both on noisy and noise-free data.

Learning Raw Image Reconstruction-Aware Deep Image Compressors

This paper examines the ability of deep image compressors to be “aware” of the additional objective of raw reconstruction and describes a general framework that enables deep networks targeting image compression to jointly consider both image fidelity errors and raw reconstruction errors.

CycleISP: Real Image Restoration via Improved Data Synthesis

This paper presents a framework that models camera imaging pipeline in forward and reverse directions that allows any number of realistic image pairs for denoising both in RAW and sRGB spaces and achieves the state-of-the-art performance on real camera benchmark datasets.

CIE XYZ Net: Unprocessing Images for Low-Level Computer Vision Tasks

This paper proposes a deep learning framework, CIE XYZ Net, that can unprocess a nonlinear image back to the canonical CIEXYZ image, which can be processed by any low-level computer vision operator and re-renderedback to the non linear image.

Single-Image HDR Reconstruction by Learning to Reverse the Camera Pipeline

This work model the HDR-to-LDR image formation pipeline as the dynamic range clipping, non-linear mapping from a camera response function, and quantization, and proposes to learn three specialized CNNs to reverse these steps.

FlexISP: a flexible camera image processing framework

  • K. Pulli
  • Computer Science
    ACM Trans. Graph.
  • 2014
This work proposes an end-to-end system that is aware of the camera and image model, enforces natural-image priors, while jointly accounting for common image processing steps like demosaicking, denoising, deconvolution, and so forth, all directly in a given output representation.

Replacing Mobile Camera ISP with a Single Deep Learning Model

PyNET is presented, a novel pyramidal CNN architecture designed for fine-grained image restoration that implicitly learns to perform all ISP steps such as image demosaicing, denoising, white balancing, color and contrast correction, demoireing, etc.

CameraNet: A Two-Stage Framework for Effective Camera ISP Learning

This paper designs a two-stage network, called CameraNet, to progressively learn the two groups of ISP tasks: restoration and enhancement, and achieves consistently compelling reconstruction quality and outperforms the recently proposed ISP learning methods.

Reconfiguring the Imaging Pipeline for Computer Vision

This work examines the role of the image signal processing (ISP) pipeline in computer vision to identify opportunities to reduce computation and save energy, and proposes a new image sensor design that can compensate for these stages.