• Corpus ID: 239998687

Deep learning via message passing algorithms based on belief propagation

  title={Deep learning via message passing algorithms based on belief propagation},
  author={Carlo Lucibello and Fabrizio Pittorino and Gabriele Perugini and Riccardo Zecchina},
Message-passing algorithms based on the Belief Propagation (BP) equations constitute a well-known distributed computational scheme. It is exact on tree-like graphical models and has also proven to be effective in many problems defined on graphs with loops (from inference to optimization, from signal processing to clustering). The BP-based scheme is fundamentally different from stochastic gradient descent (SGD), on which the current success of deep networks is based. In this paper, we present… 

Figures and Tables from this paper


Belief Propagation Neural Networks
By training BPNN-D, a learned iterative operator that provably maintains many of the desirable properties of BP for any choice of the parameters, BPNNs learns to perform the task better than the original BP: it converges 1.7x faster on Ising models while providing tighter bounds.
Mean-field message-passing equations in the Hopfield model and its generalizations.
  • M. Mézard
  • Mathematics, Medicine
    Physical review. E
  • 2017
The mean-field equations of belief-propagation and Thouless-Anderson Palmer (TAP) equations are revisited in the best understood of such machines, namely the Hopfield model of neural networks, and it is explicit how they can be used as iterative message-passing algorithms, providing a fast method to compute the local polarizations of neurons.
Understanding belief propagation and its generalizations
It is shown that BP can only converge to a fixed point that is also a stationary point of the Bethe approximation to the free energy, which enables connections to be made with variational approaches to approximate inference.
Mean-field inference methods for neural networks
A selection of classical mean-field methods and recent progress relevant for inference in neural networks are reviewed, and the principles of derivations of high-temperature expansions, the replica method and message passing algorithms are reminded, highlighting their equivalences and complementarities.
Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
This work presents a novel scalable method for learning Bayesian neural networks, called probabilistic backpropagation (PBP), which works by computing a forward propagation of probabilities through the network and then doing a backward computation of gradients.
Belief Propagation Reloaded: Learning BP-Layers for Labeling Problems
One of the simplest inference methods is taken, a truncated max-product Belief Propagation, and added what is necessary to make it a proper component of a deep learning model: connect it to learning formulations with losses on marginals and compute the backprop operation.
Inference in Deep Networks in High Dimensions
  • A. Fletcher, S. Rangan
  • Computer Science, Mathematics
    2018 IEEE International Symposium on Information Theory (ISIT)
  • 2018
The main contribution shows that the mean-squared error (MSE) of ML-VAMP can be exactly predicted in a certain large system limit and matches the Bayes optimal value recently postulated by Reeves when certain fixed point equations have unique solutions.
Vector approximate message passing
This paper considers a “vector AMP” (VAMP) algorithm and shows that VAMP has a rigorous scalar state-evolution that holds under a much broader class of large random matrices A: those that are right-rotationally invariant.
Unreasonable effectiveness of learning neural networks: From accessible states and robust ensembles to basic algorithmic schemes
It is shown that there are regions of the optimization landscape that are both robust and accessible and that their existence is crucial to achieve good performance on a class of particularly difficult learning problems, and an explanation of this good performance is proposed in terms of a nonequilibrium statistical physics framework.
Expectation Propagation for approximate Bayesian inference
  • T. Minka
  • Computer Science, Mathematics
  • 2001
Expectation Propagation approximates the belief states by only retaining expectations, such as mean and varitmce, and iterates until these expectations are consistent throughout the network, which makes it applicable to hybrid networks with discrete and continuous nodes.