# Expressive Power and Approximation Errors of Restricted Boltzmann Machines

@inproceedings{Montfar2011ExpressivePA, title={Expressive Power and Approximation Errors of Restricted Boltzmann Machines}, author={Guido Mont{\'u}far and Johannes Rauh and Nihat Ay}, booktitle={NIPS}, year={2011} }

We present explicit classes of probability distributions that can be learned by Restricted Boltzmann Machines (RBMs) depending on the number of units that they contain, and which are representative for the expressive power of the model. We use this to show that the maximal Kullback-Leibler divergence to the RBM model with n visible and m hidden units is bounded from above by (n – 1) – log(m + 1). In this way we can specify the number of hidden units that guarantees a sufficiently rich model…

## 46 Citations

### On the Expressive Power of Restricted Boltzmann Machines

- Computer ScienceNIPS
- 2013

The RBM's unnormalized log-likelihood function is characterized as a type of neural network (called an RBM network), and through a series of simulation results relate these networks to types that are better understood.

### On the Representational Efficiency of Restricted Boltzmann Machines

- Computer ScienceNIPS 2013
- 2013

The RBM's unnormalized log-likelihood function is characterized as a type of neural network, and through a series of simulation results relate these networks to ones whose representational properties are better understood.

### Geometry and expressive power of conditional restricted Boltzmann machines

- Computer ScienceJ. Mach. Learn. Res.
- 2015

This work addresses the representational power of conditional restricted Boltzmann machines, proving their ability to represent conditional Markov random fields and conditional distributions with restricted supports, the minimal size of universal approximators, the maximal model approximation errors, and on the dimension of the set of representable conditional distributions.

### Expressive Power of Conditional Restricted Boltzmann Machines

- Computer Science
- 2014

This work addresses the representational power of conditional restricted Boltzmann machine probability models, proving results on the minimal size of universal approximators of conditional probability distributions, the minimal sizes of deterministic functions, the maximal model approximation errors, and on the dimension of the set of representable conditional distributions.

### Discrete restricted Boltzmann machines

- Computer ScienceJ. Mach. Learn. Res.
- 2015

This work describes discrete restricted Boltzmann machines: probabilistic graphical models with bipartite interactions between visible and hidden discrete variables, for which these models can approximate any probability distribution on their visible states to any given accuracy.

### Universal Approximation Depth and Errors of Narrow Belief Networks with Discrete Units

- Computer ScienceNeural Computation
- 2014

This analysis covers discrete restricted Boltzmann machines and naive Bayes models as special cases and shows that a q-ary deep belief network with layers of width for some can approximate any probability distribution on without exceeding a Kullback-Leibler divergence.

### Deep Narrow Boltzmann Machines are Universal Approximators

- Computer ScienceICLR
- 2015

It is shown that deep narrow Boltzmann machines are at least as compact universal approximators as narrow sigmoid belief networks and restricted Boltz Mann machines, with respect to the currently available bounds for those models.

### Expressive Power of Conditional Restricted Boltzmann Machines for Sensorimotor Control

- Computer ScienceArXiv
- 2014

The causal structure of the sensorimotor loop is considered and the agent’s policies are represented in terms of conditional restricted Boltzmann machines (CRBMs), which can model non-trivial conditional distributions on high dimensional input-output spaces with relatively few parameters.

### Kernels and Submodels of Deep Belief Networks

- Computer Science
- 2012

This work takes the perspective of kernel transitions of distributions, which gives a unified picture of distributed representations arising from Deep Belief Networks (DBN) and other networks without lateral connections, and describes explicit classes of probability distributions that can be learned by DBNs.

### On the magnitude of parameters of RBMs being universal approximators

- Computer Science2016 23rd International Conference on Pattern Recognition (ICPR)
- 2016

For any given error and probability, a bound is provided, by which there exits an RBM computing the the probability up to the error with parameters bounded, which depends on the error and the input probability.

## References

SHOWING 1-10 OF 30 REFERENCES

### Refinements of Universal Approximation Results for Deep Belief Networks and Restricted Boltzmann Machines

- Computer ScienceNeural Computation
- 2011

It is shown that any distribution on the set of binary vectors of length can be arbitrarily well approximated by an RBM with hidden units, and this confirms a conjecture presented in Le Roux and Bengio (2010).

### Representational Power of Restricted Boltzmann Machines and Deep Belief Networks

- Computer ScienceNeural Computation
- 2008

This work proves that adding hidden units yields strictly improved modeling power, while a second theorem shows that RBMs are universal approximators of discrete distributions and suggests a new and less greedy criterion for training RBMs within DBNs.

### Restricted Boltzmann Machines are Hard to Approximately Evaluate or Simulate

- Mathematics, Computer ScienceICML
- 2010

It is established the intractability of two basic computational tasks involving RBMs, even if only a coarse approximation to the correct output is required, and there is no polynomial-time randomized algorithm for the following problem.

### Mixture Models and Representational Power of RBM ’ s , DBN ’ s and DBM ’ s

- Computer Science
- 2010

A sharp bound on the required and sufficient number of mixture components to repr esent any arbitrary visible distribution is presented and a test of universal approximat ing properties is derived and it is found that an RBM with more than2 − 1 parameters is not always auniversal approximator of distributions on {0, 1}n.

### A Practical Guide to Training Restricted Boltzmann Machines

- Computer ScienceNeural Networks: Tricks of the Trade
- 2012

This guide is an attempt to share expertise at training restricted Boltzmann machines with other machine learning researchers.

### Geometry of the Restricted Boltzmann Machine

- Computer Science, Mathematics
- 2010

A dimension formula for the tropicalized model is derived, and it is used to show that the restricted Boltzmann machine is identifiable in many cases.

### On Contrastive Divergence Learning

- Computer ScienceAISTATS
- 2005

The properties of CD learning are studied and it is shown that it provides biased estimates in general, but that the bias is typically very small.

### Deep Belief Networks Are Compact Universal Approximators

- Computer ScienceNeural Computation
- 2010

It is proved that deep but narrow feedforward neural networks with sigmoidal units can represent any Boolean expression.

### Greedy Layer-Wise Training of Deep Networks

- Computer ScienceNIPS
- 2006

These experiments confirm the hypothesis that the greedy layer-wise unsupervised training strategy mostly helps the optimization, by initializing weights in a region near a good local minimum, giving rise to internal distributed representations that are high-level abstractions of the input, bringing better generalization.

### Unsupervised Learning of Distributions of Binary Vectors Using 2-Layer Networks

- Computer ScienceNIPS
- 1991

It is shown that arbitrary distributions of binary vectors can be approximated by the combination model and shown how the weight vectors in the model can be interpreted as high order correlation patterns among the input bits, and how the combination machine can be used as a mechanism for detecting these patterns.