# Parallel convolutional processing using an integrated photonic tensor core

@article{Feldmann2021ParallelCP, title={Parallel convolutional processing using an integrated photonic tensor core}, author={Johannes Feldmann and Nathan Youngblood and Maxim Karpov and Helge Gehring and Xuan Li and Manuel Le Gallo and Xin Fu and Anton Lukashchuk and Arslan S. Raja and Junqiu Liu and C. David Wright and Abu Sebastian and Tobias J. Kippenberg and Wolfram H. P. Pernice and Harish Bhaskaran}, journal={Nature}, year={2021}, volume={589}, pages={52-58} }

With the proliferation of ultrahigh-speed mobile networks and internet-connected devices, along with the rise of artificial intelligence (AI) 1 , the world is generating exponentially increasing amounts of data that need to be processed in a fast and efficient way. Highly parallelized, fast and scalable hardware is therefore becoming progressively more important 2 . Here we demonstrate a computationally specific integrated photonic hardware accelerator (tensor core) that is capable of operating…

## 345 Citations

### Freely scalable and reconfigurable optical hardware for deep learning

- Computer Science, PhysicsScientific reports
- 2021

A digital optical neural network (DONN) with intralayer optical interconnects and reconfigurable input values with path-length-independence of optical energy consumption is proposed and it is found that digital optical data transfer is beneficial over electronics when the spacing of computational units is on the order of $$>10\,\upmu $$ > 10 μ m.

### Programmable phase-change metasurfaces on waveguides for multimode photonic convolutional neural network

- Physics, Computer ScienceNature communications
- 2021

A phase-change metasurface mode converter is demonstrated, which can be programmed to control the waveguide mode contrast, and an optical convolutional neural network is built to perform image processing tasks with high accuracy.

### Time Wavelength Interleaving Perceptron at 12 Giga-Ops/s with a Kerr Soliton Crystal Microcomb for Optical Neural Networks

- Computer Science
- 2021

By scaling the perceptron to a deep learning network using off-the-shelf telecom technology, this work can achieve high throughput operation for matrix multiplication for real-time massive data processing.

### An optical neural network using less than 1 photon per multiplication

- Physics, Computer ScienceNature communications
- 2022

The authors report an ONN with >90% accuracy image classification using <1 detected photon per scalar multiplication, and shows that optical neural networks can achieve accurate results using extremely low optical energies.

### Multi-Task Learning in Diffractive Deep Neural Networks via Hardware-Software Co-design

- Computer ScienceArXiv
- 2020

This work proposes a novel hardware-software co-design method that enables robust and noise-resilient Multi-task Learning in D$^2$2NNs and proposes a domain-specific regularization algorithm for training the proposed multi-task architecture, which can be used to flexibly adjust the desired performance for each task.

### Novel optical neural network architecture with the temporal synthetic dimension

- Physics, Computer Science
- 2021

This approach holds theibility and easiness of reconﬁguration with potentially complex functionality in achieving desired optical tasks, pointing towards promisingly perform on-chip optical computations with further miniaturization.

### Optical Stochastic Computing Architectures Using Photonic Crystal Nanocavities

- Physics, Computer ScienceArXiv
- 2021

This report proposes a transmission model considering key nanocavity device parameters, such as Quality factors, resonance wavelength and switching efficiency, and proposes the design of XOR gate and multiplexer and illustrates the use of the gates to design an edge detection filter.

### Radio-Frequency Multiply-and-Accumulate Operations with Spintronic Synapses

- Computer Science
- 2020

This work demonstrates through physical simulations with parameters extracted from exper-imental devices that frequency-multiplexed assemblies of resonators implement the corner-stone operation of artificial neural networks, the Multiply-And-Accumulate (MAC), directly on microwave inputs.

### Arbitrary linear transformations for photons in the frequency synthetic dimension

- Computer Science, PhysicsNature communications
- 2021

This work presents a photonic architecture to achieve arbitrary linear transformations by harnessing the synthetic frequency dimension of photons and shows that the same physical structure can be reconfigured to implement a wide variety of manipulations including single-frequency conversion, nonreciprocal frequency translations, and unitary as well as non-unitary transformations.

### Robust and Efficient Single-Pixel Image Classificationwith Nonlinear Optics

- Computer Science
- 2021

Nonlinear optics is poised to enable even richer and more complex operations and lift the processing capability of artificial intelligence to yet another level.

## References

SHOWING 1-10 OF 80 REFERENCES

### Large-Scale Optical Neural Networks based on Photoelectric Multiplication

- Physics, Computer SciencePhysical Review X
- 2019

Simulations of the network using models for digit- and image-classification reveal a "standard quantum limit" for optical neural networks, set by photodetector shot noise, which suggests performance below the thermodynamic limit for digital irreversible computation is theoretically possible in this device.

### Digital Electronics and Analog Photonics for Convolutional Neural Networks (DEAP-CNNs)

- Computer ScienceIEEE Journal of Selected Topics in Quantum Electronics
- 2020

A Digital Electronic and Analog Photonic (DEAP) CNN hardware architecture that has potential to be 2.8 to 14 times faster while using almost 25% less energy than current state-of-the-art graphical processing units (GPUs).

### In-memory computing on a photonic platform

- Computer ScienceScience Advances
- 2019

This work shows that integrated optics with collocated data storage and processing can be combined to enable all-photonic in-memory computations, and sets the stage for development of entirely photonic computers.

### Photonic Multiply-Accumulate Operations for Neural Networks

- Computer ScienceIEEE Journal of Selected Topics in Quantum Electronics
- 2020

This work describes the performance of photonic and electronic hardware underlying neural network models using multiply-accumulate operations, and investigates the limits of analog electronic crossbar arrays and on-chip photonic linear computing systems.

### An Optical Frontend for a Convolutional Neural Network

- PhysicsApplied optics
- 2019

An architecture that utilizes a single electrical to optical conversion by designing a free-space optical frontend unit that implements the linear operations of the first layer with the subsequent layers realized electronically.

### Calculating with light using a chip-scale all-optical abacus

- PhysicsNature Communications
- 2017

The central element of an all-optical calculator is demonstrated, a photonic abacus, which provides multistate compute-and-store operation by integrating functional phase-change materials with nanophotonic chips.

### Multipurpose silicon photonics signal processor core

- PhysicsNature Communications
- 2017

A reconfigurable but simple silicon waveguide mesh with different functionalities with a simple seven hexagonal cell structure is demonstrated, which can be applied to different fields including communications, chemical and biomedical sensing, signal processing, multiprocessor networks, and quantum information systems.

### ISAAC: A Convolutional Neural Network Accelerator with In-Situ Analog Arithmetic in Crossbars

- Computer Science2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA)
- 2016

This work explores an in-situ processing approach, where memristor crossbar arrays not only store input weights, but are also used to perform dot-product operations in an analog manner.

### Dot-product engine for neuromorphic computing: Programming 1T1M crossbar to accelerate matrix-vector multiplication

- Computer Science2016 53nd ACM/EDAC/IEEE Design Automation Conference (DAC)
- 2016

The Dot-Product Engine (DPE) is developed as a high density, high power efficiency accelerator for approximate matrix-vector multiplication, invented a conversion algorithm to map arbitrary matrix values appropriately to memristor conductances in a realistic crossbar array.

### Fast and reliable storage using a 5 bit, nonvolatile photonic memory cell

- PhysicsOptica
- 2018

This work demonstrates an optically addressed, multilevel memory capable of storing up to 34 nonvolatile reliable and repeatable levels (over 5 bits) using the phase change material Ge2Sb2Te5 integrated on a photonic waveguide and investigates the influence of write-and-erase pulse parameters on the single-pulse recrystallization, amorphization, and readout error in the memory, thus tailoring pulse properties for optimum performance.