Entropy, Free Energy, and Work of Restricted Boltzmann Machines

@article{Oh2020EntropyFE,
  title={Entropy, Free Energy, and Work of Restricted Boltzmann Machines},
  author={Sangchul Oh and Abdelkader Baggag and Hyunchul Nha},
  journal={Entropy},
  year={2020},
  volume={22}
}
A restricted Boltzmann machine is a generative probabilistic graphic network. A probability of finding the network in a certain configuration is given by the Boltzmann distribution. Given training data, its learning is done by optimizing the parameters of the energy function of the network. In this paper, we analyze the training process of the restricted Boltzmann machine in the context of statistical physics. As an illustration, for small size bar-and-stripe patterns, we calculate… 

Online Sequential Extreme Learning Machine: A New Training Scheme for Restricted Boltzmann Machines

TLDR
The proposed approach is compared to one of the well known training algorithms for Boltzmann machines named “contrastive divergence”, in term of time, accuracy and algorithmic complexity under the same conditions.

E-LANG: Energy-Based Joint Inferencing of Super and Swift Language Models

TLDR
An effective dynamic inference approach, called E-LANG, which distributes the inference between large accurate Super-models and light-weight Swift models, and can be applied to black-box pre-trained models without a need for architectural manipulations, reassembling of modules, or re-training.

EBJR: Energy-Based Joint Reasoning for Adaptive Inference

TLDR
This paper presents a new method of jointly using the large accurate models together with the small fast ones, and proposes an Energy-Based Joint Reasoning (EBJR) framework that adaptively distributes the samples between shallow and deep models to achieve an accuracy close to the deep model, but latency close toThe shallow one.

References

SHOWING 1-10 OF 44 REFERENCES

Quantum Boltzmann Machine

TLDR
This work proposes a new machine learning approach based on quantum Boltzmann distribution of a transverse-field Ising Hamiltonian that allows the QBM efficiently by sampling and discusses the possibility of using quantum annealing processors like D-Wave for QBM training and application.

Training restricted Boltzmann machines: An introduction

Number of trials required to estimate a free-energy difference, using fluctuation relations.

TLDR
The number of trials one should expect to perform is bound, using the order-∞ Rényi entropy, if one implements the "good practice" of bidirectionality, known to improve estimates of ΔF.

Equilibrium free energies from fast-switching trajectories with large time steps.

TLDR
Numerical simulations show that Newton's equation can be discretized to low order over very large time steps (limited only by the computer's ability to represent resulting values of dynamical variables) without sacrificing thermodynamic accuracy.

A high-bias, low-variance introduction to Machine Learning for physicists

A Deterministic and Generalized Framework for Unsupervised Learning with Restricted Boltzmann Machines

TLDR
This work derives a deterministic framework for the training, evaluation, and use of RBMs based upon the Thouless-Anderson-Palmer (TAP) mean-field approximation of widely-connected systems with weak interactions coming from spin-glass theory.

Entropy production fluctuation theorem and the nonequilibrium work relation for free energy differences.

  • G. Crooks
  • Economics, Physics
    Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics
  • 1999
TLDR
A generalized version of the fluctuation theorem is derived for stochastic, microscopically reversible dynamics and this generalized theorem provides a succinct proof of the nonequilibrium work relation.

Equivalence of restricted Boltzmann machines and tensor network states

TLDR
This work builds a bridge between RBM and tensor network states (TNS) widely used in quantum many-body physics research, and devise efficient algorithms to translate an RBM into the commonly used TNS.

Solving the quantum many-body problem with artificial neural networks

TLDR
A variational representation of quantum states based on artificial neural networks with a variable number of hidden neurons and a reinforcement-learning scheme that is capable of both finding the ground state and describing the unitary time evolution of complex interacting quantum systems.

How many trials should you expect to perform to estimate a free-energy difference ?

TLDR
The number of trials one should expect to perform, using the order-∞ Rényi entropy, is bound, if one implements the “good practice” of bidirectionality, known to improve estimates of ∆F .