Mutual information, neural networks and the renormalization group

@article{KochJanusz2017MutualIN,
  title={Mutual information, neural networks and the renormalization group},
  author={Maciej Koch-Janusz and Zohar Ringel},
  journal={Nature Physics},
  year={2017},
  volume={14},
  pages={578-582}
}
Physical systems differing in their microscopic details often display strikingly similar behaviour when probed at macroscopic scales. Those universal properties, largely determining their physical characteristics, are revealed by the powerful renormalization group (RG) procedure, which systematically retains ‘slow’ degrees of freedom and integrates out the rest. However, the important degrees of freedom may be difficult to identify. Here we demonstrate a machine-learning algorithm capable of… 

Relevance in the Renormalization Group and in Information Theory

TLDR
It is shown analytically that for statistical physical systems described by a field theory the relevant degrees of freedom found using IB compression indeed correspond to operators with the lowest scaling dimensions, which provides a dictionary connecting two distinct theoretical toolboxes, and an example of constructively incorporating physical interpretability in applications of deep learning in physics.

Neural Network Renormalization Group

We present a variational renormalization group (RG) approach based on a reversible generative model with hierarchical architecture. The model performs hierarchical change-of-variables transformations

Adding machine learning within Hamiltonians: Renormalization group transformations, symmetry breaking and restoration

TLDR
A physical interpretation of machine learning functions is presented, opening up the possibility to control properties of statistical systems via the inclusion of these functions in Hamiltonians, including the predictive function of a neural network as a conjugate variable coupled to an external field within the Hamiltonian of a system.

Optimal Renormalization Group Transformation from Information Theory

TLDR
This work investigates analytically the RG coarse-graining procedure and the renormalized Hamiltonian, which the RSMI algorithm defines, and shows that a perfect RSMI coarse- graining generically does not increase the range of a short-ranged Hamiltonian in any dimension.

Neural Monte Carlo renormalization group

TLDR
The results establish a solid connection between the RG transformation in physics and the deep architecture in machine learning, paving the way to further interdisciplinary research.

Entropic Dynamics in Neural Networks, the Renormalization Group and the Hamilton-Jacobi-Bellman Equation

We study the dynamics of information processing in the continuum depth limit of deep feed-forward Neural Networks (NN) and find that it can be described in language similar to the Renormalization

Entropic Dynamics for Learning in Neural Networks and the Renormalization Group

TLDR
The dynamics of information processing in the continuous depth limit of deep feed-forward Neural Networks (NN) can be described in language similar to the Renormalization Group (RG) and these algorithms can be shown to yield optimal generalization in student- teacher scenarios.

Towards quantifying information flows: relative entropy in deep neural networks and the renormalization group

TLDR
The analogy between the renormalization group (RG) and deep neural networks, wherein subsequent layers of neurons are analogous to successive steps along the RG, is investigated, and the monotonic increase confirms the connection between the relative entropy and the c-theorem.

Is Deep Learning a Renormalization Group Flow?

TLDR
It is argued that correlation functions between hidden and visible neurons are capable of diagnosing RG-like coarse graining and comparing a single layer of a deep network to a single step in the RG flow.

Maximum Multiscale Entropy and Neural Network Regularization

TLDR
This paper presents different ways of generalizing the maximum entropy result by incorporating the notion of scale, and shows that in a teacher-student scenario, the multiscale Gibbs posterior can achieve a smaller excess risk than the single-scale Gibbs posterior.
...

References

SHOWING 1-10 OF 42 REFERENCES

The renormalization group via statistical inference

In physics, one attempts to infer the rules governing a system given only the results of imperfect measurements. Hence, microscopic theories may be effectively indistinguishable experimentally. We

Parameter Space Compression Underlies Emergent Theories and Predictive Models

TLDR
An information-theoretical approach is used to distinguish the important parameters in two archetypical physics models and traces the emergence of an effective theory for long-scale observables to a compression of the parameter space quantified by the eigenvalues of the Fisher Information Matrix.

An exact mapping between the Variational Renormalization Group and Deep Learning

TLDR
This work constructs an exact mapping from the variational renormalization group, first introduced by Kadanoff, and deep learning architectures based on Restricted Boltzmann Machines (RBMs), and suggests that deep learning algorithms may be employing a generalized RG-like scheme to learn relevant features from data.

Learning phase transitions by confusion

TLDR
This work proposes a neural-network approach to finding phase transitions, based on the performance of a neural network after it is trained with data that are deliberately labelled incorrectly, and paves the way to the development of a generic tool for identifying unexplored phase transitions.

Solving the quantum many-body problem with artificial neural networks

TLDR
A variational representation of quantum states based on artificial neural networks with a variable number of hidden neurons and a reinforcement-learning scheme that is capable of both finding the ground state and describing the unitary time evolution of complex interacting quantum systems.

Detecting hidden spatial and spatio-temporal structures in glasses and complex physical systems by multiresolution network clustering

TLDR
A general method for characterizing the “natural” structures in complex physical systems via multi-scale network analysis based on “community detection” and identifies the dominant structures (disjoint or overlapping) and general length scales by analyzing extrema of the information theory measures.

The renormalization group: Critical phenomena and the Kondo problem

This review covers several topics involving renormalization group ideas. The solution of the $s$-wave Kondo Hamiltonian, describing a single magnetic impurity in a nonmagnetic metal, is explained in

Learning Thermodynamics with Boltzmann Machines

TLDR
A Boltzmann machine is developed that is capable of modeling thermodynamic observables for physical systems in thermal equilibrium and can faithfully reproduce the observables of the physical system.

Why Does Deep and Cheap Learning Work So Well?

TLDR
It is argued that when the statistical process generating the data is of a certain hierarchical form prevalent in physics and machine learning, a deep neural network can be more efficient than a shallow one.