Tighter PAC-Bayes Generalisation Bounds by Leveraging Example Difficulty

@article{Biggs2022TighterPG,
  title={Tighter PAC-Bayes Generalisation Bounds by Leveraging Example Difficulty},
  author={Felix Biggs and Benjamin Guedj},
  journal={ArXiv},
  year={2022},
  volume={abs/2210.11289}
}
We introduce a modified version of the excess risk, which can be used to obtain tighter, fast-rate PAC-Bayesian generalisation bounds. This modified excess risk leverages information about the relative hardness of data examples to reduce the variance of its empirical counterpart, tightening the bound. We combine this with a new bound for [ − 1 , 1]-valued (and potentially non-independent) signed losses, which is more favourable when they empirically have low variance around 0. The primary new… 
1 Citations

Tables from this paper

Streaming PAC-Bayes Gaussian process regression with a performance guarantee for online decision making

A new theoretical framework for online GPs based on the online probably approximately correct (PAC) Bayes theory is presented, which offers both a guarantee of generalized performance and good accuracy and performs well empirically on several regression datasets.

References

SHOWING 1-10 OF 35 REFERENCES

PAC-Bayes Unexpected Bernstein Inequality

A new PAC-Bayesian generalization bound is presented, which can be expected to converge to 0 faster whenever a Bernstein/Tsybakov condition holds, thus connecting PAC- Bayesiangeneralization and {\em excess risk\/} bounds.

On the role of data in PAC-Bayes bounds

This work shows that the bound based on the oracle prior can be suboptimal, and applies this new principle in the setting of nonconvex learning, simulating data-dependent oracle priors on MNIST and Fashion MNIST with and without held-out data, and demonstrating new nonvacuous bounds in both cases.

Learning Stochastic Majority Votes by Minimizing a PAC-Bayes Generalization Bound

The resulting stochastic majority vote learning algorithm achieves state-of-the-art accuracy and benefits from (non-vacuous) tight generalization bounds, in a series of numerical experiments when compared to competing algorithms which also minimize PACBayes objectives.

PAC-Bayesian Inequalities for Martingales

The results extend the PAC-Bayesian analysis in learning theory from the i.i.d. setting to martingales opening the way for its application to importance weighted sampling, reinforcement learning, and other interactive learning domains, as well as many other domains in probability theory and statistics, where Martingales are encountered.

Entropy-SGD optimizes the prior of a PAC-Bayes bound: Data-dependent PAC-Bayes priors via differential privacy

We show that Entropy-SGD (Chaudhari et al., 2017), when viewed as a learning algorithm, optimizes a PAC-Bayes bound on the risk of a Gibbs (posterior) classifier, i.e., a randomized classifier

A Note on the Efficient Evaluation of PAC-Bayes Bounds

This manuscript presents a very general alternative which makes computational savings on the order of the dataset size.

User-friendly introduction to PAC-Bayes bounds

This paper describes a simplified version of the localization technique of [34, 36] that was missed by the community, and later rediscovered as “mutual information bounds” and is an attempt to provide an elementary introduction to PAC-Bayes theory.

Differentiable PAC–Bayes Objectives with Partially Aggregated Neural Networks

This work reformulates a PAC–Bayesian bound for signed-output networks to derive in combination with the above a directly optimisable, differentiable objective and a generalisation guarantee, without using a surrogate loss or loosening the bound.

Second Order PAC-Bayesian Bounds for the Weighted Majority Vote

We present a novel analysis of the expected risk of weighted majority vote in multiclass classification. The analysis takes correlation of predictions by ensemble members into account and provides a

PAC-bayes bounds with data dependent priors

The experimental work illustrates that the new bounds can be significantly tighter than the original PAC-Bayes bound when applied to SVMs, and among them the combination of the prior PAC- Bayes bound and the prior SVM algorithm gives the tightest bound.