Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimates
@inproceedings{Negrea2019InformationTheoreticGB, title={Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimates}, author={Jeffrey Negrea and Mahdi Haghifam and G. Dziugaite and Ashish Khisti and Daniel M. Roy}, booktitle={NeurIPS}, year={2019} }
In this work, we improve upon the stepwise analysis of noisy iterative learning algorithms initiated by Pensia, Jog, and Loh (2018) and recently extended by Bu, Zou, and Veeravalli (2019). Our main contributions are significantly improved mutual information bounds for Stochastic Gradient Langevin Dynamics via data-dependent estimates. Our approach is based on the variational characterization of mutual information and the use of data-dependent priors that forecast the mini-batch gradient based… CONTINUE READING
Supplemental Presentations
Figures and Topics from this paper
Figures
22 Citations
Sharpened Generalization Bounds based on Conditional Mutual Information and an Application to Noisy, Iterative Algorithms
- Computer Science, Mathematics
- NeurIPS
- 2020
- 8
- PDF
Nonvacuous Loss Bounds with Fast Rates for Neural Networks via Conditional Information Measures
- Computer Science, Mathematics
- ArXiv
- 2020
- 1
- Highly Influenced
- PDF
Tightening Mutual Information-Based Bounds on Generalization Error
- Computer Science, Mathematics
- IEEE Journal on Selected Areas in Information Theory
- 2020
- 3
- PDF
Generalization Bounds via Information Density and Conditional Information Density
- Computer Science, Mathematics
- IEEE Journal on Selected Areas in Information Theory
- 2020
- 9
- PDF
Shape Matters: Understanding the Implicit Bias of the Noise Covariance
- Computer Science, Mathematics
- ArXiv
- 2020
- 5
- PDF
Tightening Mutual Information Based Bounds on Generalization Error
- Computer Science
- 2019 IEEE International Symposium on Information Theory (ISIT)
- 2019
- 28
Reasoning About Generalization via Conditional Mutual Information
- Computer Science, Mathematics
- COLT
- 2020
- 14
- PDF
Information-Theoretic Generalization Bounds for Meta-Learning and Applications
- Medicine, Computer Science
- Entropy
- 2021
- 4
- PDF
References
SHOWING 1-10 OF 36 REFERENCES
On Generalization Error Bounds of Noisy Gradient Methods for Non-Convex Learning
- Computer Science, Mathematics
- ICLR
- 2020
- 10
- PDF
Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpoints
- Mathematics, Computer Science
- COLT
- 2018
- 53
- Highly Influential
- PDF
Generalization Error Bounds for Noisy, Iterative Algorithms
- Computer Science, Mathematics
- 2018 IEEE International Symposium on Information Theory (ISIT)
- 2018
- 25
- Highly Influential
- PDF
Entropy-SGD optimizes the prior of a PAC-Bayes bound: Data-dependent PAC-Bayes priors via differential privacy
- Computer Science, Mathematics
- NeurIPS
- 2018
- 46
- PDF
Train faster, generalize better: Stability of stochastic gradient descent
- Computer Science, Mathematics
- ICML
- 2016
- 524
- PDF
Tightening Mutual Information Based Bounds on Generalization Error
- Computer Science
- 2019 IEEE International Symposium on Information Theory (ISIT)
- 2019
- 28
- Highly Influential
Chaining Mutual Information and Tightening Generalization Bounds
- Computer Science, Mathematics
- NeurIPS
- 2018
- 34
- PDF
Information-theoretic analysis of stability and bias of learning algorithms
- Computer Science
- 2016 IEEE Information Theory Workshop (ITW)
- 2016
- 36
- PDF