Corpus ID: 202773824

Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimates

@inproceedings{Negrea2019InformationTheoreticGB,
  title={Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimates},
  author={Jeffrey Negrea and Mahdi Haghifam and G. Dziugaite and Ashish Khisti and Daniel M. Roy},
  booktitle={NeurIPS},
  year={2019}
}
  • Jeffrey Negrea, Mahdi Haghifam, +2 authors Daniel M. Roy
  • Published in NeurIPS 2019
  • Computer Science, Mathematics
    • In this work, we improve upon the stepwise analysis of noisy iterative learning algorithms initiated by Pensia, Jog, and Loh (2018) and recently extended by Bu, Zou, and Veeravalli (2019). Our main contributions are significantly improved mutual information bounds for Stochastic Gradient Langevin Dynamics via data-dependent estimates. Our approach is based on the variational characterization of mutual information and the use of data-dependent priors that forecast the mini-batch gradient based… CONTINUE READING
    View PDF on arXiv
    22 Citations
    Highly Influencial Citations
    2
    Background Citations
    15
    Methods Citations
    10
    Results Citations
    1

    Supplemental Presentations

    Presentation Slides
    Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimates
    Explore Further
    Discover more papers related to the topics discussed in this paper

    Figures and Topics from this paper

    Figures

    Explore Further: Topics Discussed in This Paper

    22 Citations
    Citation Type

    Citation Type

    Sharpened Generalization Bounds based on Conditional Mutual Information and an Application to Noisy, Iterative Algorithms
    • 8
    • PDF
    Nonvacuous Loss Bounds with Fast Rates for Neural Networks via Conditional Information Measures
    • 1
    • Highly Influenced
    • PDF
    Tightening Mutual Information-Based Bounds on Generalization Error
    • 3
    • PDF
    Generalization Bounds via Information Density and Conditional Information Density
    • 9
    • PDF
    Shape Matters: Understanding the Implicit Bias of the Noise Covariance
    • 5
    • PDF
    Tightening Mutual Information Based Bounds on Generalization Error
    • 28
    On the role of data in PAC-Bayes bounds
    • 7
    • PDF
    Reasoning About Generalization via Conditional Mutual Information
    • 14
    • PDF
    Recent advances in deep learning theory
    • 4
    • PDF
    Information-Theoretic Generalization Bounds for Meta-Learning and Applications
    • 4
    • PDF

    References

    SHOWING 1-10 OF 36 REFERENCES
    On Generalization Error Bounds of Noisy Gradient Methods for Non-Convex Learning
    • 10
    • PDF
    Information matrices and generalization
    • 11
    On Variational Bounds of Mutual Information
    • 131
    • PDF
    Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpoints
    • 53
    • Highly Influential
    • PDF
    Generalization Error Bounds for Noisy, Iterative Algorithms
    • 25
    • Highly Influential
    • PDF
    Entropy-SGD optimizes the prior of a PAC-Bayes bound: Data-dependent PAC-Bayes priors via differential privacy
    • 46
    • PDF
    Train faster, generalize better: Stability of stochastic gradient descent
    • 524
    • PDF
    Tightening Mutual Information Based Bounds on Generalization Error
    • 28
    • Highly Influential
    Chaining Mutual Information and Tightening Generalization Bounds
    • 34
    • PDF
    Information-theoretic analysis of stability and bias of learning algorithms
    • 36
    • PDF