# Entropy, Free Energy, and Work of Restricted Boltzmann Machines

@article{Oh2020EntropyFE, title={Entropy, Free Energy, and Work of Restricted Boltzmann Machines}, author={Sangchul Oh and Abdelkader Baggag and Hyunchul Nha}, journal={Entropy}, year={2020}, volume={22} }

A restricted Boltzmann machine is a generative probabilistic graphic network. A probability of finding the network in a certain configuration is given by the Boltzmann distribution. Given training data, its learning is done by optimizing the parameters of the energy function of the network. In this paper, we analyze the training process of the restricted Boltzmann machine in the context of statistical physics. As an illustration, for small size bar-and-stripe patterns, we calculate…

## Figures from this paper

## 3 Citations

### Online Sequential Extreme Learning Machine: A New Training Scheme for Restricted Boltzmann Machines

- Computer Science
- 2020

The proposed approach is compared to one of the well known training algorithms for Boltzmann machines named “contrastive divergence”, in term of time, accuracy and algorithmic complexity under the same conditions.

### E-LANG: Energy-Based Joint Inferencing of Super and Swift Language Models

- Computer ScienceACL
- 2022

An effective dynamic inference approach, called E-LANG, which distributes the inference between large accurate Super-models and light-weight Swift models, and can be applied to black-box pre-trained models without a need for architectural manipulations, reassembling of modules, or re-training.

### EBJR: Energy-Based Joint Reasoning for Adaptive Inference

- Computer ScienceBMVC
- 2021

This paper presents a new method of jointly using the large accurate models together with the small fast ones, and proposes an Energy-Based Joint Reasoning (EBJR) framework that adaptively distributes the samples between shallow and deep models to achieve an accuracy close to the deep model, but latency close toThe shallow one.

## References

SHOWING 1-10 OF 44 REFERENCES

### Quantum Boltzmann Machine

- Computer Science, Physics
- 2016

This work proposes a new machine learning approach based on quantum Boltzmann distribution of a transverse-field Ising Hamiltonian that allows the QBM efficiently by sampling and discusses the possibility of using quantum annealing processors like D-Wave for QBM training and application.

### Training restricted Boltzmann machines: An introduction

- Computer SciencePattern Recognit.
- 2014

### Number of trials required to estimate a free-energy difference, using fluctuation relations.

- PhysicsPhysical review. E
- 2016

The number of trials one should expect to perform is bound, using the order-∞ Rényi entropy, if one implements the "good practice" of bidirectionality, known to improve estimates of ΔF.

### Equilibrium free energies from fast-switching trajectories with large time steps.

- PhysicsThe Journal of chemical physics
- 2006

Numerical simulations show that Newton's equation can be discretized to low order over very large time steps (limited only by the computer's ability to represent resulting values of dynamical variables) without sacrificing thermodynamic accuracy.

### A high-bias, low-variance introduction to Machine Learning for physicists

- Computer Science, PhysicsPhysics reports
- 2019

### A Deterministic and Generalized Framework for Unsupervised Learning with Restricted Boltzmann Machines

- Computer SciencePhysical Review X
- 2018

This work derives a deterministic framework for the training, evaluation, and use of RBMs based upon the Thouless-Anderson-Palmer (TAP) mean-field approximation of widely-connected systems with weak interactions coming from spin-glass theory.

### Entropy production fluctuation theorem and the nonequilibrium work relation for free energy differences.

- Economics, PhysicsPhysical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics
- 1999

A generalized version of the fluctuation theorem is derived for stochastic, microscopically reversible dynamics and this generalized theorem provides a succinct proof of the nonequilibrium work relation.

### Equivalence of restricted Boltzmann machines and tensor network states

- Computer Science
- 2018

This work builds a bridge between RBM and tensor network states (TNS) widely used in quantum many-body physics research, and devise efficient algorithms to translate an RBM into the commonly used TNS.

### Solving the quantum many-body problem with artificial neural networks

- Computer Science, PhysicsScience
- 2017

A variational representation of quantum states based on artificial neural networks with a variable number of hidden neurons and a reinforcement-learning scheme that is capable of both finding the ground state and describing the unitary time evolution of complex interacting quantum systems.

### How many trials should you expect to perform to estimate a free-energy difference ?

- Biology
- 2016

The number of trials one should expect to perform, using the order-∞ Rényi entropy, is bound, if one implements the “good practice” of bidirectionality, known to improve estimates of ∆F .