# Training Restricted Boltzmann Machine via the Thouless-Anderson-Palmer free energy

@inproceedings{Gabri2015TrainingRB, title={Training Restricted Boltzmann Machine via the Thouless-Anderson-Palmer free energy}, author={Marylou Gabri{\'e} and Eric W. Tramel and Florent Krzakala}, booktitle={NIPS}, year={2015} }

Restricted Boltzmann machines are undirected neural networks which have been shown to be effective in many applications, including serving as initializations for training deep multi-layer neural networks. One of the main reasons for their success is the existence of efficient and practical stochastic algorithms, such as contrastive divergence, for unsupervised training. We propose an alternative deterministic iterative procedure based on an improved mean field method from statistical physics…

## 36 Citations

A Deterministic and Generalized Framework for Unsupervised Learning with Restricted Boltzmann Machines

- Mathematics, Computer SciencePhysical Review X
- 2018

This work derives a deterministic framework for the training, evaluation, and use of RBMs based upon the Thouless-Anderson-Palmer (TAP) mean-field approximation of widely-connected systems with weak interactions coming from spin-glass theory.

Spectral dynamics of learning in restricted Boltzmann machines

- Computer Science
- 2017

A generic statistical ensemble is proposed for the weight matrix of the RBM and its mean evolution is characterized, unveiling in some way how the selected modes interact in later stages of the learning procedure and defining a deterministic learning curve for the R BM.

Mean-field inference methods for neural networks

- Physics, Computer ScienceArXiv
- 2019

A selection of classical mean-field methods and recent progress relevant for inference in neural networks are reviewed, and the principles of derivations of high-temperature expansions, the replica method and message passing algorithms are reminded, highlighting their equivalences and complementarities.

Mean-field message-passing equations in the Hopfield model and its generalizations.

- Mathematics, MedicinePhysical review. E
- 2017

The mean-field equations of belief-propagation and Thouless-Anderson Palmer (TAP) equations are revisited in the best understood of such machines, namely the Hopfield model of neural networks, and it is explicit how they can be used as iterative message-passing algorithms, providing a fast method to compute the local polarizations of neurons.

Boltzmann machine learning with a variational quantum algorithm

- Physics, Computer Science
- 2020

This work proposes a method to implement the Boltzmann machine learning by using Noisy Intermediate-Scale Quantum (NISQ) devices, prepares an initial pure state that contains all possible computational basis states with the same amplitude, and applies a variational imaginary time simulation.

Boltzmann Machines as Generalized Hopfield Networks: A Review of Recent Results and Outlooks

- Computer Science, MedicineEntropy
- 2021

Interestingly, the Boltzmann machine and the Hopfield network, if considered to be two cognitive processes (learning and information retrieval), are nothing more than two sides of the same coin and it is possible to exactly map the one into the other.

Inverse problems for structured datasets using parallel TAP equations and restricted Boltzmann machines

- MedicineScientific reports
- 2021

An efficient algorithm to solve inverse problems in the presence of binary clustered datasets based on the estimation of the posterior using the Thouless–Anderson–Palmer (TAP) equations in a parallel updating scheme and can be applied to large system sizes.

Neighborhood-Based Stopping Criterion for Contrastive Divergence

- Computer ScienceIEEE Trans. Neural Networks Learn. Syst.
- 2018

This manuscript presents a simple and cheap alternative to the reconstruction error, based on the inclusion of information contained in neighboring states to the training set, as a stopping criterion for CD learning.

From Boltzmann Machines to Neural Networks and Back Again

- Computer Science, MathematicsNeurIPS
- 2020

This work gives new results for learning Restricted Boltzmann Machines, probably the most well-studied class of latent variable models, and gives an algorithm for learning a natural class of supervised RBMs with better runtime than what is possible for its related class of networks without distributional assumptions.

Restricted Boltzmann Machine with Multivalued Hidden Variables: a model suppressing over-fitting

- Computer Science, MathematicsRev. Socionetwork Strateg.
- 2019

This study proposes an RBM with multivalued hidden variables, which is a simple extension of conventional RBMs and demonstrates that the proposed model is better than the conventional model via numerical experiments for contrastive divergence learning with artificial data and a classification problem with MNIST.

## References

SHOWING 1-10 OF 36 REFERENCES

Deep Boltzmann Machines

- Computer ScienceAISTATS
- 2009

A new learning algorithm for Boltzmann machines that contain many layers of hidden variables that is made more efficient by using a layer-by-layer “pre-training” phase that allows variational inference to be initialized with a single bottomup pass.

Joint Training Deep Boltzmann Machines for Classification

- Computer Science, MathematicsICLR
- 2013

This work introduces a new method for training deep Boltzmann machines jointly, and shows that this approach performs competitively for classification and outperforms previous methods in terms of accuracy of approximate inference and classification with missing inputs.

The limitations of deterministic Boltzmann machine learning

- Mathematics
- 1993

The stochastic Boltzmann machine (SBM) learning procedure allows a system of stochastic binary units at thermal equilibrium to model arbitrary probabilistic distributions of binary vectors, but the…

Deterministic Boltzmann Learning Performs Steepest Descent in Weight-Space

- Mathematics, Computer ScienceNeural Computation
- 1989

By using the appropriate interpretation for the way in which a DBM represents the probability of an output vector given an input vector, it is shown that the DBM performs steepest descent in the same function as the original SBM, except at rare discontinuities.

Classification using discriminative restricted Boltzmann machines

- Computer ScienceICML '08
- 2008

This paper presents an evaluation of different learning algorithms for RBMs which aim at introducing a discriminative component to RBM training and improve their performance as classifiers, and demonstrates how discriminating RBMs can also be successfully employed in a semi-supervised setting.

Inductive Principles for Restricted Boltzmann Machine Learning

- Mathematics, Computer ScienceAISTATS
- 2010

This paper studies learning methods for binary restricted Boltzmann machines based on ratio matching and generalized score matching and compares them to a range of existing learning methods including stochastic maximum likelihood, contrastive divergence, and pseudo-likelihood.

Boltzmann Machine Learning Using Mean Field Theory and Linear Response Correction

- Mathematics, Computer ScienceNIPS
- 1997

A new approximate learning algorithm for Boltzmann Machines is presented, using a systematic expansion of the Gibbs free energy to second order in the weights, which concludes that the inclusion of diagonal weights improves the Weiss approximation for paramagnetic problems, but not for spin glass problems.

A Practical Guide to Training Restricted Boltzmann Machines

- Computer ScienceNeural Networks: Tricks of the Trade
- 2012

This guide is an attempt to share expertise at training restricted Boltzmann machines with other machine learning researchers.

Training restricted Boltzmann machines using approximations to the likelihood gradient

- Mathematics, Computer ScienceICML '08
- 2008

A new algorithm for training Restricted Boltzmann Machines is introduced, which is compared to some standard Contrastive Divergence and Pseudo-Likelihood algorithms on the tasks of modeling and classifying various types of data.

A Fast Learning Algorithm for Deep Belief Nets

- Mathematics, Computer ScienceNeural Computation
- 2006

A fast, greedy algorithm is derived that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory.