Learned Smartphone ISP on Mobile NPUs with Deep Learning, Mobile AI 2021 Challenge: Report

@article{Ignatov2021LearnedSI,
  title={Learned Smartphone ISP on Mobile NPUs with Deep Learning, Mobile AI 2021 Challenge: Report},
  author={Andrey D. Ignatov and Cheng-Ming Chiang and Hsien-Kai Kuo and Anastasia Sycheva and R. Timofte and Min-Hung Chen and Man-Yu Lee and Yu-Syuan Xu and Yu Tseng and Shusong Xu and Jin Guo and Chao-Hung Chen and Ming-Chun Hsyu and Wen-Chia Tsai and Chao-Wei Chen and Grigory Malivenko and Minsu Kwon and Myungje Lee and Jaeyoon Yoo and Changbeom Kang and Shinjo Wang and Zheng Shaolong and Hao Dejun and Xie Fen and Feng Zhuang and Yipeng Ma and Jingyang Peng and Tao Wang and Fenglong Song and Chih-Chung Hsu and Kwan-Lin Chen and Mei Wu and Vishal M. Chudasama and Kalpesh P. Prajapati and Heena Patel and Anjali Sarvaiya and K. Upla and K. Raja and Raghavendra Ramachandra and C. Busch and Etienne de Stoutz},
  journal={2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)},
  year={2021},
  pages={2503-2514}
}
As the quality of mobile cameras starts to play a crucial role in modern smartphones, more and more attention is now being paid to ISP algorithms used to improve various perceptual aspects of mobile photos. In this Mobile AI challenge, the target was to develop an end-to-end deep learning-based image signal processing (ISP) pipeline that can replace classical hand-crafted ISPs and achieve nearly real-time performance on smartphone NPUs. For this, the participants were provided with a novel… Expand
Fast Camera Image Denoising on Mobile GPUs with Deep Learning, Mobile AI 2021 Challenge: Report
  • Andrey D. Ignatov, Kim Byeoung-su, +29 authors Feifei Chen
  • Computer Science, Engineering
  • 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
  • 2021
TLDR
The first Mobile AI challenge is introduced, where the target is to develop an end-to-end deep learning-based image denoising solution that can demonstrate high efficiency on smartphone GPUs. Expand
Fast and Accurate Quantized Camera Scene Detection on Smartphones, Mobile AI 2021 Challenge: Report
TLDR
The first Mobile AI challenge is introduced, where the target is to develop quantized deep learning-based camera scene classification solutions that can demonstrate a real-time performance on smartphones and IoT platforms. Expand
Real-Time Video Super-Resolution on Smartphones with Deep Learning, Mobile AI 2021 Challenge: Report
TLDR
The target is to develop an end-to-end deep learning-based video super-resolution solutions that can achieve a realtime performance on mobile GPUs and can upscale videos to HD resolution at up to 80 FPS while demonstrating high fidelity results. Expand
Real-Time Quantized Image Super-Resolution on Mobile NPUs, Mobile AI 2021 Challenge: Report
  • Andrey D. Ignatov, R. Timofte, +20 authors Shengpeng Wang
  • Computer Science, Engineering
  • 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
  • 2021
TLDR
The first Mobile AI challenge is introduced, where the target is to develop an end-to-end deep learning-based image super-resolution solutions that can demonstrate a real-time performance on mobile or edge NPUs. Expand
Fast and Accurate Single-Image Depth Estimation on Mobile Devices, Mobile AI 2021 Challenge: Report
TLDR
This paper introduces the first Mobile AI challenge, where the target is to develop an end-to-end deep learning-based depth estimation solutions that can demonstrate a nearly realtime performance on smartphones and IoT platforms. Expand
CSANet: High Speed Channel Spatial Attention Network for Mobile ISP
The Image Signal Processor (ISP) is a customized device to restore RGB images from the pixel signals of CMOS image sensor. In order to realize this function, a series of processing units areExpand
Fast and Accurate Camera Scene Detection on Smartphones
TLDR
A novel Camera Scene Detection Dataset (CamSDD) containing more than 11K manually crawled images belonging to 30 different scene categories is proposed and an efficient and NPU-friendly CNN model is proposed that demonstrates a top-3 accuracy of 99.5% on this dataset and achieves more than 200 FPS on the recent mobile SoCs. Expand

References

SHOWING 1-10 OF 58 REFERENCES
Fast Camera Image Denoising on Mobile GPUs with Deep Learning, Mobile AI 2021 Challenge: Report
  • Andrey D. Ignatov, Kim Byeoung-su, +29 authors Feifei Chen
  • Computer Science, Engineering
  • 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
  • 2021
TLDR
The first Mobile AI challenge is introduced, where the target is to develop an end-to-end deep learning-based image denoising solution that can demonstrate high efficiency on smartphone GPUs. Expand
Fast and Accurate Quantized Camera Scene Detection on Smartphones, Mobile AI 2021 Challenge: Report
TLDR
The first Mobile AI challenge is introduced, where the target is to develop quantized deep learning-based camera scene classification solutions that can demonstrate a real-time performance on smartphones and IoT platforms. Expand
Real-Time Video Super-Resolution on Smartphones with Deep Learning, Mobile AI 2021 Challenge: Report
TLDR
The target is to develop an end-to-end deep learning-based video super-resolution solutions that can achieve a realtime performance on mobile GPUs and can upscale videos to HD resolution at up to 80 FPS while demonstrating high fidelity results. Expand
Real-Time Quantized Image Super-Resolution on Mobile NPUs, Mobile AI 2021 Challenge: Report
  • Andrey D. Ignatov, R. Timofte, +20 authors Shengpeng Wang
  • Computer Science, Engineering
  • 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
  • 2021
TLDR
The first Mobile AI challenge is introduced, where the target is to develop an end-to-end deep learning-based image super-resolution solutions that can demonstrate a real-time performance on mobile or edge NPUs. Expand
Fast and Accurate Single-Image Depth Estimation on Mobile Devices, Mobile AI 2021 Challenge: Report
TLDR
This paper introduces the first Mobile AI challenge, where the target is to develop an end-to-end deep learning-based depth estimation solutions that can demonstrate a nearly realtime performance on smartphones and IoT platforms. Expand
Replacing Mobile Camera ISP with a Single Deep Learning Model
TLDR
PyNET is presented, a novel pyramidal CNN architecture designed for fine-grained image restoration that implicitly learns to perform all ISP steps such as image demosaicing, denoising, white balancing, color and contrast correction, demoireing, etc. Expand
PyNET-CA: Enhanced PyNET with Channel Attention for End-to-End Mobile Image Signal Processing
TLDR
The model enhances PyNET, a recently proposed state-of-the-art model for mobile ISP, and improves its performance with channel attention and subpixel reconstruction module, and demonstrates the performance of the proposed method with comparative experiments and results from the AIM 2020 learned smartphone ISP challenge. Expand
AWNet: Attentive Wavelet Network for Image ISP
TLDR
A novel network that utilizes the attention mechanism and wavelet transform, dubbed AWNet, to tackle this learnable image ISP problem, and enables us to restore favorable image details from RAW information and achieve a larger receptive field while remaining high efficiency in terms of computational cost. Expand
PIRM Challenge on Perceptual Image Enhancement on Smartphones: Report
TLDR
This paper reviews the first challenge on efficient perceptual image enhancement with the focus on deploying deep learning models on smartphones and proposes solutions that significantly improved baseline results defining the state-of-the-art for image enhancement on smartphones. Expand
CSANet: High Speed Channel Spatial Attention Network for Mobile ISP
The Image Signal Processor (ISP) is a customized device to restore RGB images from the pixel signals of CMOS image sensor. In order to realize this function, a series of processing units areExpand
...
1
2
3
4
5
...