Robustness Implies Generalization via Data-Dependent Generalization Bounds

@article{Kawaguchi2022RobustnessIG,
  title={Robustness Implies Generalization via Data-Dependent Generalization Bounds},
  author={Kenji Kawaguchi and Zhun Deng and Kyle Luh and Jiaoyang Huang},
  journal={ArXiv},
  year={2022},
  volume={abs/2206.13497}
}
This paper proves that robustness implies generalization via data-dependent generalization bounds. As a result, robustness and generalization are shown to be connected closely in a data-dependent manner. Our bounds improve previous bounds in two directions, to solve an open problem that has seen little development since 2010. The first is to reduce the dependence on the covering number. The second is to remove the dependence on the hypothesis space. We present several examples, including ones… 

Figures from this paper

References

SHOWING 1-10 OF 56 REFERENCES

Weak Convergence and Empirical Processes: With Applications to Statistics

This chapter discusses Convergence: Weak, Almost Uniform, and in Probability, which focuses on the part of Convergence of the Donsker Property which is concerned with Uniformity and Metrization.

Concentration Inequalities for Multinoulli Random Variables

This work investigates concentration inequalities for Dirichlet and Multinomial random variables and focuses on p̂n ∼ 1 nMultinomial(n, p), a bounded random variable in [0, D].

Reading Digits in Natural Images with Unsupervised Feature Learning

A new benchmark dataset for research use is introduced containing over 600,000 labeled digits cropped from Street View images, and variants of two recently proposed unsupervised feature learning methods are employed, finding that they are convincingly superior on benchmarks.

Stability and Generalization

These notions of stability for learning algorithms are defined and it is shown how to use these notions to derive generalization error bounds based on the empirical error and the leave-one-out error.

Robustness and generalization

We derive generalization bounds for learning algorithms based on their robustness: the property that if a testing sample is “similar” to a training sample, then the testing error is close to the

Robustness and Regularization of Support Vector Machines

This work considers regularized support vector machines and shows that they are precisely equivalent to a new robust optimization formulation, thus establishing robustness as the reason regularized SVMs generalize well and gives a new proof of consistency of (kernelized) SVMs.

The Equivalence of Weak, Strong and Complete Convergence in $L_1$ for Kernel Density Estimates

THEOREM 1. Let K be a nonnegative Borel measurable function on Rd with f K(x) dx = 1. Then the following conditions are equivalent: (i) J -* 0 in probability as n -oo, some f; (ii) Jn -O 0 in

GradientBased Learning Applied to Document Recognition

Various methods applied to handwritten character recognition are reviewed and compared and Convolutional Neural Networks, that are specifically designed to deal with the variability of 2D shapes, are shown to outperform all other techniques.

Deep Learning for Classical Japanese Literature

This work introduces Kuz Kushiji-MNIST, a dataset which focuses on Kuzushiji (cursive Japanese), as well as two larger, more challenging datasets, KuzUSHiji-49 and Kuzushaiji-Kanji, which are intended to engage the machine learning community into the world of classical Japanese literature.

Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms

Fashion-MNIST is intended to serve as a direct drop-in replacement for the original MNIST dataset for benchmarking machine learning algorithms, as it shares the same image size, data format and the structure of training and testing splits.
...