Reconfiguring the Imaging Pipeline for Computer Vision

@article{Buckler2017ReconfiguringTI,
  title={Reconfiguring the Imaging Pipeline for Computer Vision},
  author={Mark Buckler and Suren Jayasuriya and Adrian Sampson},
  journal={2017 IEEE International Conference on Computer Vision (ICCV)},
  year={2017},
  pages={975-984}
}
Advancements in deep learning have ignited an explosion of research on efficient hardware for embedded computer vision. [] Key Method We use eight computer vision algorithms and a reversible pipeline simulation tool to study the imaging system's impact on vision performance. For both CNN-based and classical vision algorithms, we observe that only two ISP stages, demosaicing and gamma compression, are critical for task performance. We propose a new image sensor design that can compensate for these stages.
Deep learning for camera data acquisition, control, and image estimation
TLDR
The development of artificial neural networks and their recent intersection with computational imaging is reviewed, and in more detail how deep learning impacts the primary strategies of computational photography: focal plane modulation, lens design, and robotic control is considered.
EVA²: Exploiting Temporal Redundancy in Live Computer Vision
TLDR
A new algorithm, activation motion compensation, detects changes in the visual input and incrementally updates a previously-computed activation and applies well-known motion estimation techniques to adapt to visual changes to avoid unnecessary computation on most frames.
Minimalistic Image Signal Processing for Deep Learning Applications
TLDR
To make in-sensor accelerators practical, energy-efficient operations that yield most of the benefits of an ISP and reduce covariate shift between the training (ISP processed images) and target (RAW images) distributions are described.
ISP4ML: The Role of Image Signal Processing in Efficient Deep Learning Vision Systems
TLDR
This work builds software models of a configurable ISP and an imaging sensor to train CNNs on ImageNet with a range of different ISP settings and functionality and outlines the system-level trade-offs between prediction accuracy and computational cost.
ISP4ML: Understanding the Role of Image Signal Processing in Efficient Deep Learning Vision Systems
TLDR
This work builds software models of a configurable ISP and an imaging sensor in order to train CNNs on ImageNet with a range of different ISP settings and functionality, and outlines the system-level trade-offs between prediction accuracy and computational cost.
Euphrates: Algorithm-SoC Co-Design for Low-Power Mobile Continuous Vision
TLDR
An algorithm-architecture co-designed system, Euphrates, that simultaneously improves the energyefficiency and performance of continuous vision tasks and co-optimizes different SoC IP blocks in the vision pipeline collectively.
VisionISP: Repurposing the Image Signal Processor for Computer Vision Applications
TLDR
Experimental results show that VisionISP boosts the performance of a subsequent computer vision system trained to detect objects in an autonomous driving setting and significantly reduces data transmission needs by reducing the bit-depth and resolution while preserving the relevant information.
Dirty Pixels: Towards End-to-end Image Processing and Perception
TLDR
This work proposes an end-to-end differentiable architecture that jointly performs demosaicking, denoising, deblurring, tone-mapping, and classification and shows that state-of-the-art ISPs discard information that is essential in corner cases, where conventional imaging and perception stacks fail.
Hardware-in-the-Loop End-to-End Optimization of Camera Image Processing Pipelines
TLDR
A hardware-in-the-loop method that directly optimizes hardware image processing pipelines for end-to-end domain-specific losses by solving a nonlinear multi-objective optimization problem with a novel 0th-order stochastic solver directly interfaced with the hardware ISP is presented.
End-to-end High Dynamic Range Camera Pipeline Optimization
TLDR
The proposed HDR ISP hyperparameter search is formulated as an end-to-end optimization problem, proposing a mixed 0th and 1st-order block coordinate descent optimizer that jointly learns sensor, ISP and detector network weights using RAW image data augmented with emulated SNR transition region artifacts.
...
...

References

SHOWING 1-10 OF 54 REFERENCES
ASP Vision: Optically Computing the First Layer of Convolutional Neural Networks Using Angle Sensitive Pixels
TLDR
ASPs replace both image sensing and the first layer of a conventional CNN by directly performing optical edge filtering, saving sensing energy, data bandwidth, and CNN FLOPS to compute, while achieving similar performance compared to traditional deep learning pipelines.
RedEye: Analog ConvNet Image Sensor Architecture for Continuous Mobile Vision
TLDR
The design of RedEye is designed to mitigate analog design complexity, using a modular column-parallel design to promote physical design reuse and algorithmic cyclic reuse and programmable mechanisms to admit noise for tunable energy reduction.
Evaluating programmable architectures for imaging and vision applications
TLDR
This work compares flexible ISP architectures, using applications written in the Darkroom image processing language, and considers several optimizations on these two base architectures, such as register file partitioning for SIMD, bus based routing and pipelined wires for CGRA, and line buffer variations.
ShiDianNao: Shifting vision processing closer to the sensor
TLDR
This paper proposes an accelerator which is 60x more energy efficient than the previous state-of-the-art neural network accelerator, designed down to the layout at 65 nm, with a modest footprint and consuming only 320 mW, but still about 30x faster than high-end GPUs.
Dirty Pixels: Optimizing Image Classification Architectures for Raw Sensor Data
TLDR
The proposed architecture dramatically improves the accuracy of a classification network in low light and other challenging conditions, outperforming alternative approaches such as retraining the network on noisy and blurry images and preprocessing raw sensor inputs with conventional denoising and deblurring algorithms.
Rigel: flexible multi-rate image processing hardware
TLDR
This paper presents Rigel, which takes pipelines specified in the new multi-rate architecture and lowers them to FPGA implementations, and demonstrates depth from stereo, Lucas-Kanade, the SIFT descriptor, and a Gaussian pyramid running on two FPGAs.
Energy characterization and optimization of image sensing toward continuous mobile vision
TLDR
This work reports the first publicly known experimental and analytical characterization of CMOS image sensors, finding that modern image sensors are not energy-proportional, and proposes architectural modifications to the image sensor that would further improve operational efficiency.
Learning Sensor Multiplexing Design through Back-propagation
TLDR
This paper learns the camera sensor's color multiplexing pattern by encoding it as layer whose learnable weights determine which color channel will be measured at each location, and jointly trained with a reconstruction network that operates on the corresponding sensor measurements to produce a full color image.
A New In-Camera Imaging Model for Color Computer Vision and Its Application
TLDR
A major limitation of the imaging model employed in conventional radiometric calibration methods is found and a new in-camera imaging model is proposed that fits well with today's cameras and is significantly more accurate than any existing methods.
Ultra-low-power image signal processor for smart camera applications
TLDR
This chapter starts with the intuition that the perceptive quality of images is not strongly correlated with the accuracy of object detection algorithms and proposes three techniques that require only minor modifications to the baseline ISP but dramatically reduce the ISP energy consumption in object detection mode for smart camera applications.
...
...