Data-Dependent Stability of Stochastic Gradient Descent

@inproceedings{Kuzborskij2018DataDependentSO,
  title={Data-Dependent Stability of Stochastic Gradient Descent},
  author={Ilja Kuzborskij and Christoph H. Lampert},
  booktitle={ICML},
  year={2018}
}
We establish a data-dependent notion of algorithmic stability for Stochastic Gradient Descent (SGD) and employ it to develop novel generalization bounds. This is in contrast to previous distribution-free algorithmic stability results for SGD which depend on the worstcase constants. By virtue of the data-dependent argument, our bounds provide new insights into learning with SGD on convex and non-convex problems. In the convex case, we show that the bound on the generalization error is… CONTINUE READING