A connection between probability, physics and neural networks

@article{Ranftl2022ACB,
  title={A connection between probability, physics and neural networks},
  author={Sascha Ranftl},
  journal={ArXiv},
  year={2022},
  volume={abs/2209.12737}
}
We illustrate an approach that can be exploited for constructing neural networks which a priori obey physical laws. We start with a simple single-layer neural network (NN) but refrain from choosing the activation functions yet. Under certain conditions and in the infinite-width limit, we may apply the central limit theorem, upon which the NN output becomes Gaussian. We may then investigate and manipulate the limit network by falling back on Gaussian process (GP) theory. It is observed that… 

Figures from this paper

Gaussian Process Priors for Systems of Linear Partial Differential Equations with Constant Coefficients

The Ehrenpreis-Palamodov fundamental principle is applied, which works like a non-linear Fourier transform, to construct GP kernels mirroring standard spectral methods for GPs, and can infer probable solutions of linear PDE systems from any data such as noisy measurements, or pointwise defined initial and boundary conditions.

Probabilistic Analysis of an RL Circuit Transient Response under Inductor Failure Conditions

We apply probability theory for the analysis of the exponentially decaying transient response of a resistor inductor electric circuit with partially known value of the inductance due to its failure.

References

SHOWING 1-10 OF 42 REFERENCES

Deep Neural Networks as Gaussian Processes

The exact equivalence between infinitely wide deep networks and GPs is derived and it is found that test performance increases as finite-width trained networks are made wider and more similar to a GP, and thus that GP predictions typically outperform those of finite- width networks.

Physical Symmetries Embedded in Neural Networks

The focus of the present work is to embed physical constraints into the structure of the neural network to address the second fundamental challenge of neural networks lack of interpretability and physics-agnostic design.

Scientific Machine Learning Through Physics–Informed Neural Networks: Where we are and What’s Next

This article provides a comprehensive review of the literature on PINNs and indicates that most research has focused on customizing the PINN through different activation functions, gradient optimization techniques, neural network structures, and loss function structures.

Invariance of Weight Distributions in Rectified MLPs

The Central Limit Theorem is used to show that for certain activation functions, kernels corresponding to layers with weight distributions having $0$ mean and finite absolute third moment are asymptotically universal, and are well approximated by the kernel corresponding to layer with spherical Gaussian weights.

Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes

This work derives an analogous equivalence for multi-layer convolutional neural networks (CNNs) both with and without pooling layers, and introduces a Monte Carlo method to estimate the GP corresponding to a given neural network architecture, even in cases where the analytic form has too many terms to be computationally feasible.

Computing with Infinite Networks

For neural networks with a wide class of weight-priors, it can be shown that in the limit of an infinite number of hidden units the prior over functions tends to a Gaussian process. In this paper

Infinite attention: NNGP and NTK for deep attention networks

A rigorous extension of results to NNs involving attention layers is provided, showing that unlike single- head attention, which induces non-Gaussian behaviour, multi-head attention architectures behave as GPs as the number of heads tends to infinity.

Expressive Priors in Bayesian Neural Networks: Kernel Combinations and Periodic Functions

The derived BNN architectures provide a principled approach to designing BNNs that incorporate prior knowledge about a function and are shown to produce periodic kernels, which are often useful in this context.

Gaussian Processes for Data Fulfilling Linear Differential Equations

  • C. Albert
  • Mathematics, Computer Science
    Proceedings
  • 2019
A method to reconstruct fields, source strengths and physical parameters based on Gaussian process regression is presented for the case where data are known to fulfill a given linear differential