A tutorial on stochastic approximation algorithms for training Restricted Boltzmann Machines and Deep Belief Nets

@article{Swersky2010ATO,
  title={A tutorial on stochastic approximation algorithms for training Restricted Boltzmann Machines and Deep Belief Nets},
  author={Kevin Swersky and Bo Chen and Benjamin M. Marlin and Nando de Freitas},
  journal={2010 Information Theory and Applications Workshop (ITA)},
  year={2010},
  pages={1-10}
}
In this study, we provide a direct comparison of the Stochastic Maximum Likelihood algorithm and Contrastive Divergence for training Restricted Boltzmann Machines using the MNIST data set. We demonstrate that Stochastic Maximum Likelihood is superior when using the Restricted Boltzmann Machine as a classifier, and that the algorithm can be greatly improved using the technique of iterate averaging from the field of stochastic approximation. We further show that training with optimal parameters… CONTINUE READING
Highly Cited
This paper has 73 citations. REVIEW CITATIONS

Citations

Publications citing this paper.
Showing 1-10 of 34 extracted citations

Restricted Boltzmann Machines and Deep Belief Networks on Sunway Cluster

2016 IEEE 18th International Conference on High Performance Computing and Communications; IEEE 14th International Conference on Smart City; IEEE 2nd International Conference on Data Science and Systems (HPCC/SmartCity/DSS) • 2016
View 4 Excerpts
Highly Influenced

Practical recommendations for gradient-based training of deep architectures

Neural Networks: Tricks of the Trade • 2012
View 7 Excerpts
Highly Influenced

Restricted Boltzmann Machines and Deep Belief Networks on multi-core processors

The 2012 International Joint Conference on Neural Networks (IJCNN) • 2012
View 4 Excerpts
Highly Influenced

Adiabatic Persistent Contrastive Divergence learning

2017 IEEE International Symposium on Information Theory (ISIT) • 2017
View 1 Excerpt

Fuzzy restricted Boltzmann machine and deep belief network: A comparison on image reconstruction

2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC) • 2017
View 1 Excerpt

How to Center Deep Boltzmann Machines

Journal of Machine Learning Research • 2016
View 1 Excerpt

74 Citations

01020'11'13'15'17'19
Citations per Year
Semantic Scholar estimates that this publication has 74 citations based on the available data.

See our FAQ for additional information.

References

Publications referenced by this paper.
Showing 1-10 of 37 references

Training Products of Experts by Minimizing Contrastive Divergence

Neural Computation • 2002
View 10 Excerpts
Highly Influenced

The MNIST database of handwritt en digits

Y. LeCun, C. Cortes
NEC Research Institute, http://yann. lecun. com/exdb/mni st/i dex. html .
View 7 Excerpts
Highly Influenced

Training a deep autoen coder or a classifier on MNIST digits — source code

R. Salakhutdinov, G. Hinton
2006. • 2006
View 3 Excerpts
Highly Influenced

Semantic hashing

Int. J. Approx. Reasoning • 2009
View 1 Excerpt

The difficulty of training deep architectures and the effect of u ns pervised pre-training

D. Erhan, P. Manzagol, Y. Bengio, S. Bengio, P. Vince nt
AISTATS, 2009. • 2009
View 1 Excerpt

A direct proof that the true RBM parameters a re a fixed point of both ML and CD1 in the asymptotic setting

B. Marlin
2008. • 2008
View 1 Excerpt

A tutorial on adaptive MCMC

Statistics and Computing • 2008
View 1 Excerpt

Similar Papers

Loading similar papers…