# Tighter PAC-Bayes Generalisation Bounds by Leveraging Example Difficulty

@article{Biggs2022TighterPG,
title={Tighter PAC-Bayes Generalisation Bounds by Leveraging Example Difficulty},
author={Felix Biggs and Benjamin Guedj},
journal={ArXiv},
year={2022},
volume={abs/2210.11289}
}
• Published 20 October 2022
• Computer Science
• ArXiv
We introduce a modiﬁed version of the excess risk, which can be used to obtain tighter, fast-rate PAC-Bayesian generalisation bounds. This modiﬁed excess risk leverages information about the relative hardness of data examples to reduce the variance of its empirical counterpart, tightening the bound. We combine this with a new bound for [ − 1 , 1]-valued (and potentially non-independent) signed losses, which is more favourable when they empirically have low variance around 0. The primary new…
1 Citations

## Tables from this paper

• Computer Science
ArXiv
• 2022
A new theoretical framework for online GPs based on the online probably approximately correct (PAC) Bayes theory is presented, which offers both a guarantee of generalized performance and good accuracy and performs well empirically on several regression datasets.

## References

SHOWING 1-10 OF 35 REFERENCES

• Computer Science
• 2019
A new PAC-Bayesian generalization bound is presented, which can be expected to converge to 0 faster whenever a Bernstein/Tsybakov condition holds, thus connecting PAC- Bayesiangeneralization and {\em excess risk\/} bounds.
• Computer Science
AISTATS
• 2021
This work shows that the bound based on the oracle prior can be suboptimal, and applies this new principle in the setting of nonconvex learning, simulating data-dependent oracle priors on MNIST and Fashion MNIST with and without held-out data, and demonstrating new nonvacuous bounds in both cases.
• Computer Science
NeurIPS
• 2021
The resulting stochastic majority vote learning algorithm achieves state-of-the-art accuracy and benefits from (non-vacuous) tight generalization bounds, in a series of numerical experiments when compared to competing algorithms which also minimize PACBayes objectives.
• Mathematics
IEEE Transactions on Information Theory
• 2012
The results extend the PAC-Bayesian analysis in learning theory from the i.i.d. setting to martingales opening the way for its application to importance weighted sampling, reinforcement learning, and other interactive learning domains, as well as many other domains in probability theory and statistics, where Martingales are encountered.
• Computer Science
NeurIPS
• 2018
We show that Entropy-SGD (Chaudhari et al., 2017), when viewed as a learning algorithm, optimizes a PAC-Bayes bound on the risk of a Gibbs (posterior) classifier, i.e., a randomized classifier
This manuscript presents a very general alternative which makes computational savings on the order of the dataset size.
This paper describes a simplified version of the localization technique of [34, 36] that was missed by the community, and later rediscovered as “mutual information bounds” and is an attempt to provide an elementary introduction to PAC-Bayes theory.
• Computer Science
Entropy
• 2021
This work reformulates a PAC–Bayesian bound for signed-output networks to derive in combination with the above a directly optimisable, differentiable objective and a generalisation guarantee, without using a surrogate loss or loosening the bound.
• Computer Science
NeurIPS
• 2020
We present a novel analysis of the expected risk of weighted majority vote in multiclass classification. The analysis takes correlation of predictions by ensemble members into account and provides a
• Computer Science
J. Mach. Learn. Res.
• 2012
The experimental work illustrates that the new bounds can be significantly tighter than the original PAC-Bayes bound when applied to SVMs, and among them the combination of the prior PAC- Bayes bound and the prior SVM algorithm gives the tightest bound.