# Deep Neural Networks Motivated by Partial Differential Equations

@article{Ruthotto2019DeepNN, title={Deep Neural Networks Motivated by Partial Differential Equations}, author={Lars Ruthotto and E. Haber}, journal={Journal of Mathematical Imaging and Vision}, year={2019}, volume={62}, pages={352-364} }

Partial differential equations (PDEs) are indispensable for modeling many physical phenomena and also commonly used for solving image processing tasks. In the latter area, PDE-based approaches interpret image data as discretizations of multivariate functions and the output of image processing algorithms as solutions to certain PDEs. Posing image processing problems in the infinite-dimensional setting provides powerful tools for their analysis and solution. For the last few decades, the… Expand

#### Supplemental Code

Github Repo

Via Papers with Code

A pytorch toolbox designed for experimentation using a generalization of the Resnet design.

#### 195 Citations

FOCNet: A Fractional Optimal Control Network for Image Denoising

- Computer Science
- 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- 2019

An advanced image denoising network, namely FOCNet, is developed by solving a fractional optimal control (FOC) problem and the network structure is designed based on the discretization of an fractional-order differential equation, which enjoys long-term memory in both forward and backward passes. Expand

Review: Ordinary Differential Equations For Deep Learning

- Computer Science, Mathematics
- ArXiv
- 2019

This paper formulates the optimization problem where a traditional neural network is replaced by a continuous model (ODE) and discusses how to utilize the optimization methods that are popular in optimal control literature to help the training of machine learning problems. Expand

Quantized convolutional neural networks through the lens of partial differential equations

- Computer Science
- ArXiv
- 2021

Quantization of Convolutional Neural Networks (CNNs) is a common approach to ease the computational burden involved in the deployment of CNNs, especially on low-resource edge devices. However,… Expand

Low-Cost Parameterizations of Deep Convolution Neural Networks

- Mathematics, Computer Science
- ArXiv
- 2018

This paper proposes new architectures that use a sparser coupling between the channels and thereby reduce both the number of trainable weights and the computational cost of the CNN. Expand

Designing Rotationally Invariant Neural Networks from PDEs and Variational Methods

- Computer Science, Mathematics
- ArXiv
- 2021

Partial differential equation (PDE) models and their associated variational energy formulations are often rotationally invariant by design. This ensures that a rotation of the input results in a… Expand

Fractional Deep Neural Network via Constrained Optimization

- Computer Science, Mathematics
- Mach. Learn. Sci. Technol.
- 2021

A novel algorithmic framework for a deep neural network (DNN), which in a mathematically rigorous manner, allows us to incorporate history (or memory) into the network -- it ensures all layers are connected to one another. Expand

Networks for Nonlinear Diffusion Problems in Imaging

- Medicine, Computer Science
- Journal of Mathematical Imaging and Vision
- 2019

This work develops a network architecture based on nonlinear diffusion processes, named DiffNet, that is well suited for diffusion-related problems in imaging and obtains competitive results to the established U-Net architecture. Expand

PDE-GCN: Novel Architectures for Graph Neural Networks Motivated by Partial Differential Equations

- Computer Science
- ArXiv
- 2021

This work proposes a family of architectures that are motivated by numerical methods for solving Partial Differential Equations on manifolds, and can generalize and be effective for various types of problems from different fields. Expand

An Energy Approach to the Solution of Partial Differential Equations in Computational Mechanics via Machine Learning: Concepts, Implementation and Applications

- Computer Science, Mathematics
- ArXiv
- 2019

This contribution focuses in mechanical problems and analyze the energetic format of the PDE, where the energy of a mechanical system seems to be the natural loss function for a machine learning method to approach a mechanical problem. Expand

Identification of kernels in a convolutional neural network: connections between the level set equation and deep learning for image segmentation

- Medicine, Computer Science
- Medical Imaging: Image Processing
- 2020

Which convolution filters are present in a standard CNN are analyzed, to see whether finite difference stencils are learned during training, and certain patterns that form at certain layers in the network, where the learned CNN kernels depart from known convolution kernels used to solve the level set equation are observed. Expand

#### References

SHOWING 1-10 OF 65 REFERENCES

Learning across scales - A multiscale method for Convolution Neural Networks

- Computer Science
- AAAI
- 2018

This work shows that the forward propagation in CNNs can be interpreted as a time-dependent nonlinear differential equation and learning as controlling the parameters of the differential equation such that the network approximates the data-label relation for given training data. Expand

Anisotropic diffusion in image processing

- Computer Science, Mathematics
- 1996

This work states that all scale-spaces fulllling a few fairly natural axioms are governed by parabolic PDEs with the original image as initial condition, which means that, if one image is brighter than another, then this order is preserved during the entire scale-space evolution. Expand

Deep relaxation: partial differential equations for optimizing deep neural networks

- Computer Science, Mathematics
- ArXiv
- 2017

Stochastic homogenization theory allows us to better understand the convergence of the algorithm, and a stochastic control interpretation is used to prove that a modified algorithm converges faster than SGD in expectation. Expand

Trainable Nonlinear Reaction Diffusion: A Flexible Framework for Fast and Effective Image Restoration

- Computer Science, Medicine
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- 2017

This work proposes a dynamic nonlinear reaction diffusion model with time-dependent parameters, which preserves the structural simplicity of diffusion models and take only a small number of diffusion steps, which makes the inference procedure extremely fast. Expand

Stable Architectures for Deep Neural Networks

- Computer Science, Mathematics
- ArXiv
- 2017

New forward propagation techniques inspired by systems of Ordinary Differential Equations (ODE) are proposed that overcome this challenge and lead to well-posed learning problems for arbitrarily deep networks. Expand

Partial differential equations for training deep neural networks

- Computer Science
- 2017 51st Asilomar Conference on Signals, Systems, and Computers
- 2017

This paper establishes a connection between non-convex optimization and nonlinear partial differential equations (PDEs) and suggests new algorithms based on the non-viscous Hamilton-Jacobi PDE that can effectively tackle the high dimensionality of modern neural networks. Expand

Numerical Methods for Evolutionary Differential Equations

- Computer Science
- 2008

Researchers who simulate processes that are modeled by evolutionary differential equations will find material on the principles underlying the appropriate method to use and the pitfalls that accompany each method useful. Expand

Reversible Architectures for Arbitrarily Deep Residual Neural Networks

- Computer Science, Mathematics
- AAAI
- 2018

From this interpretation, a theoretical framework on stability and reversibility of deep neural networks is developed, and three reversible neural network architectures that can go arbitrarily deep in theory are derived. Expand

ImageNet classification with deep convolutional neural networks

- Computer Science
- Commun. ACM
- 2012

A large, deep convolutional neural network was trained to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes and employed a recently developed regularization method called "dropout" that proved to be very effective. Expand

Maximum Principle Based Algorithms for Deep Learning

- Computer Science, Mathematics
- J. Mach. Learn. Res.
- 2017

The continuous dynamical system approach to deep learning is explored in order to devise alternative frameworks for training algorithms using the Pontryagin's maximum principle, demonstrating that it obtains favorable initial convergence rate per-iteration, provided Hamiltonian maximization can be efficiently carried out. Expand