• Corpus ID: 10943585

Training Restricted Boltzmann Machine via the Thouless-Anderson-Palmer free energy

@inproceedings{Gabri2015TrainingRB,
  title={Training Restricted Boltzmann Machine via the Thouless-Anderson-Palmer free energy},
  author={Marylou Gabri{\'e} and Eric W. Tramel and Florent Krzakala},
  booktitle={NIPS},
  year={2015}
}
Restricted Boltzmann machines are undirected neural networks which have been shown to be effective in many applications, including serving as initializations for training deep multi-layer neural networks. One of the main reasons for their success is the existence of efficient and practical stochastic algorithms, such as contrastive divergence, for unsupervised training. We propose an alternative deterministic iterative procedure based on an improved mean field method from statistical physics… 
A Deterministic and Generalized Framework for Unsupervised Learning with Restricted Boltzmann Machines
TLDR
This work derives a deterministic framework for the training, evaluation, and use of RBMs based upon the Thouless-Anderson-Palmer (TAP) mean-field approximation of widely-connected systems with weak interactions coming from spin-glass theory.
Spectral dynamics of learning in restricted Boltzmann machines
TLDR
A generic statistical ensemble is proposed for the weight matrix of the RBM and its mean evolution is characterized, unveiling in some way how the selected modes interact in later stages of the learning procedure and defining a deterministic learning curve for the R BM.
Mean-field inference methods for neural networks
TLDR
A selection of classical mean-field methods and recent progress relevant for inference in neural networks are reviewed, and the principles of derivations of high-temperature expansions, the replica method and message passing algorithms are reminded, highlighting their equivalences and complementarities.
Mean-field message-passing equations in the Hopfield model and its generalizations.
  • M. Mézard
  • Mathematics, Medicine
    Physical review. E
  • 2017
TLDR
The mean-field equations of belief-propagation and Thouless-Anderson Palmer (TAP) equations are revisited in the best understood of such machines, namely the Hopfield model of neural networks, and it is explicit how they can be used as iterative message-passing algorithms, providing a fast method to compute the local polarizations of neurons.
Boltzmann machine learning with a variational quantum algorithm
TLDR
This work proposes a method to implement the Boltzmann machine learning by using Noisy Intermediate-Scale Quantum (NISQ) devices, prepares an initial pure state that contains all possible computational basis states with the same amplitude, and applies a variational imaginary time simulation.
Boltzmann Machines as Generalized Hopfield Networks: A Review of Recent Results and Outlooks
TLDR
Interestingly, the Boltzmann machine and the Hopfield network, if considered to be two cognitive processes (learning and information retrieval), are nothing more than two sides of the same coin and it is possible to exactly map the one into the other.
Inverse problems for structured datasets using parallel TAP equations and restricted Boltzmann machines
TLDR
An efficient algorithm to solve inverse problems in the presence of binary clustered datasets based on the estimation of the posterior using the Thouless–Anderson–Palmer (TAP) equations in a parallel updating scheme and can be applied to large system sizes.
Neighborhood-Based Stopping Criterion for Contrastive Divergence
TLDR
This manuscript presents a simple and cheap alternative to the reconstruction error, based on the inclusion of information contained in neighboring states to the training set, as a stopping criterion for CD learning.
From Boltzmann Machines to Neural Networks and Back Again
TLDR
This work gives new results for learning Restricted Boltzmann Machines, probably the most well-studied class of latent variable models, and gives an algorithm for learning a natural class of supervised RBMs with better runtime than what is possible for its related class of networks without distributional assumptions.
Restricted Boltzmann Machine with Multivalued Hidden Variables: a model suppressing over-fitting
TLDR
This study proposes an RBM with multivalued hidden variables, which is a simple extension of conventional RBMs and demonstrates that the proposed model is better than the conventional model via numerical experiments for contrastive divergence learning with artificial data and a classification problem with MNIST.
...
1
2
3
4
...

References

SHOWING 1-10 OF 36 REFERENCES
Deep Boltzmann Machines
TLDR
A new learning algorithm for Boltzmann machines that contain many layers of hidden variables that is made more efficient by using a layer-by-layer “pre-training” phase that allows variational inference to be initialized with a single bottomup pass.
Joint Training Deep Boltzmann Machines for Classification
TLDR
This work introduces a new method for training deep Boltzmann machines jointly, and shows that this approach performs competitively for classification and outperforms previous methods in terms of accuracy of approximate inference and classification with missing inputs.
The limitations of deterministic Boltzmann machine learning
The stochastic Boltzmann machine (SBM) learning procedure allows a system of stochastic binary units at thermal equilibrium to model arbitrary probabilistic distributions of binary vectors, but the
Deterministic Boltzmann Learning Performs Steepest Descent in Weight-Space
TLDR
By using the appropriate interpretation for the way in which a DBM represents the probability of an output vector given an input vector, it is shown that the DBM performs steepest descent in the same function as the original SBM, except at rare discontinuities.
Classification using discriminative restricted Boltzmann machines
TLDR
This paper presents an evaluation of different learning algorithms for RBMs which aim at introducing a discriminative component to RBM training and improve their performance as classifiers, and demonstrates how discriminating RBMs can also be successfully employed in a semi-supervised setting.
Inductive Principles for Restricted Boltzmann Machine Learning
TLDR
This paper studies learning methods for binary restricted Boltzmann machines based on ratio matching and generalized score matching and compares them to a range of existing learning methods including stochastic maximum likelihood, contrastive divergence, and pseudo-likelihood.
Boltzmann Machine Learning Using Mean Field Theory and Linear Response Correction
TLDR
A new approximate learning algorithm for Boltzmann Machines is presented, using a systematic expansion of the Gibbs free energy to second order in the weights, which concludes that the inclusion of diagonal weights improves the Weiss approximation for paramagnetic problems, but not for spin glass problems.
A Practical Guide to Training Restricted Boltzmann Machines
TLDR
This guide is an attempt to share expertise at training restricted Boltzmann machines with other machine learning researchers.
Training restricted Boltzmann machines using approximations to the likelihood gradient
TLDR
A new algorithm for training Restricted Boltzmann Machines is introduced, which is compared to some standard Contrastive Divergence and Pseudo-Likelihood algorithms on the tasks of modeling and classifying various types of data.
A Fast Learning Algorithm for Deep Belief Nets
TLDR
A fast, greedy algorithm is derived that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory.
...
1
2
3
4
...