Deep Neural Networks Motivated by Partial Differential Equations

@article{Ruthotto2019DeepNN,
  title={Deep Neural Networks Motivated by Partial Differential Equations},
  author={Lars Ruthotto and E. Haber},
  journal={Journal of Mathematical Imaging and Vision},
  year={2019},
  volume={62},
  pages={352-364}
}
  • Lars Ruthotto, E. Haber
  • Published 2019
  • Computer Science, Mathematics
  • Journal of Mathematical Imaging and Vision
Partial differential equations (PDEs) are indispensable for modeling many physical phenomena and also commonly used for solving image processing tasks. In the latter area, PDE-based approaches interpret image data as discretizations of multivariate functions and the output of image processing algorithms as solutions to certain PDEs. Posing image processing problems in the infinite-dimensional setting provides powerful tools for their analysis and solution. For the last few decades, the… Expand
FOCNet: A Fractional Optimal Control Network for Image Denoising
TLDR
An advanced image denoising network, namely FOCNet, is developed by solving a fractional optimal control (FOC) problem and the network structure is designed based on the discretization of an fractional-order differential equation, which enjoys long-term memory in both forward and backward passes. Expand
Review: Ordinary Differential Equations For Deep Learning
TLDR
This paper formulates the optimization problem where a traditional neural network is replaced by a continuous model (ODE) and discusses how to utilize the optimization methods that are popular in optimal control literature to help the training of machine learning problems. Expand
Quantized convolutional neural networks through the lens of partial differential equations
Quantization of Convolutional Neural Networks (CNNs) is a common approach to ease the computational burden involved in the deployment of CNNs, especially on low-resource edge devices. However,Expand
Low-Cost Parameterizations of Deep Convolution Neural Networks
TLDR
This paper proposes new architectures that use a sparser coupling between the channels and thereby reduce both the number of trainable weights and the computational cost of the CNN. Expand
Designing Rotationally Invariant Neural Networks from PDEs and Variational Methods
Partial differential equation (PDE) models and their associated variational energy formulations are often rotationally invariant by design. This ensures that a rotation of the input results in aExpand
Fractional Deep Neural Network via Constrained Optimization
TLDR
A novel algorithmic framework for a deep neural network (DNN), which in a mathematically rigorous manner, allows us to incorporate history (or memory) into the network -- it ensures all layers are connected to one another. Expand
Networks for Nonlinear Diffusion Problems in Imaging
TLDR
This work develops a network architecture based on nonlinear diffusion processes, named DiffNet, that is well suited for diffusion-related problems in imaging and obtains competitive results to the established U-Net architecture. Expand
PDE-GCN: Novel Architectures for Graph Neural Networks Motivated by Partial Differential Equations
TLDR
This work proposes a family of architectures that are motivated by numerical methods for solving Partial Differential Equations on manifolds, and can generalize and be effective for various types of problems from different fields. Expand
An Energy Approach to the Solution of Partial Differential Equations in Computational Mechanics via Machine Learning: Concepts, Implementation and Applications
TLDR
This contribution focuses in mechanical problems and analyze the energetic format of the PDE, where the energy of a mechanical system seems to be the natural loss function for a machine learning method to approach a mechanical problem. Expand
Identification of kernels in a convolutional neural network: connections between the level set equation and deep learning for image segmentation
TLDR
Which convolution filters are present in a standard CNN are analyzed, to see whether finite difference stencils are learned during training, and certain patterns that form at certain layers in the network, where the learned CNN kernels depart from known convolution kernels used to solve the level set equation are observed. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 65 REFERENCES
Learning across scales - A multiscale method for Convolution Neural Networks
TLDR
This work shows that the forward propagation in CNNs can be interpreted as a time-dependent nonlinear differential equation and learning as controlling the parameters of the differential equation such that the network approximates the data-label relation for given training data. Expand
Anisotropic diffusion in image processing
TLDR
This work states that all scale-spaces fulllling a few fairly natural axioms are governed by parabolic PDEs with the original image as initial condition, which means that, if one image is brighter than another, then this order is preserved during the entire scale-space evolution. Expand
Deep relaxation: partial differential equations for optimizing deep neural networks
TLDR
Stochastic homogenization theory allows us to better understand the convergence of the algorithm, and a stochastic control interpretation is used to prove that a modified algorithm converges faster than SGD in expectation. Expand
Trainable Nonlinear Reaction Diffusion: A Flexible Framework for Fast and Effective Image Restoration
  • Yunjin Chen, T. Pock
  • Computer Science, Medicine
  • IEEE Transactions on Pattern Analysis and Machine Intelligence
  • 2017
TLDR
This work proposes a dynamic nonlinear reaction diffusion model with time-dependent parameters, which preserves the structural simplicity of diffusion models and take only a small number of diffusion steps, which makes the inference procedure extremely fast. Expand
Stable Architectures for Deep Neural Networks
TLDR
New forward propagation techniques inspired by systems of Ordinary Differential Equations (ODE) are proposed that overcome this challenge and lead to well-posed learning problems for arbitrarily deep networks. Expand
Partial differential equations for training deep neural networks
TLDR
This paper establishes a connection between non-convex optimization and nonlinear partial differential equations (PDEs) and suggests new algorithms based on the non-viscous Hamilton-Jacobi PDE that can effectively tackle the high dimensionality of modern neural networks. Expand
Numerical Methods for Evolutionary Differential Equations
TLDR
Researchers who simulate processes that are modeled by evolutionary differential equations will find material on the principles underlying the appropriate method to use and the pitfalls that accompany each method useful. Expand
Reversible Architectures for Arbitrarily Deep Residual Neural Networks
TLDR
From this interpretation, a theoretical framework on stability and reversibility of deep neural networks is developed, and three reversible neural network architectures that can go arbitrarily deep in theory are derived. Expand
ImageNet classification with deep convolutional neural networks
TLDR
A large, deep convolutional neural network was trained to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes and employed a recently developed regularization method called "dropout" that proved to be very effective. Expand
Maximum Principle Based Algorithms for Deep Learning
TLDR
The continuous dynamical system approach to deep learning is explored in order to devise alternative frameworks for training algorithms using the Pontryagin's maximum principle, demonstrating that it obtains favorable initial convergence rate per-iteration, provided Hamiltonian maximization can be efficiently carried out. Expand
...
1
2
3
4
5
...