Corpus ID: 53387011

On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length

@article{Jastrzebski2019OnTR,
  title={On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length},
  author={Stanislaw Jastrzebski and Zachary Kenton and Nicolas Ballas and Asja Fischer and Yoshua Bengio and A. Storkey},
  journal={arXiv: Machine Learning},
  year={2019}
}

Stanislaw Jastrzebski, Zachary Kenton, +3 authors A. Storkey

Published 2019

Mathematics, Computer Science

arXiv: Machine Learning

Stochastic Gradient Descent (SGD) based training of neural networks with a large learning rate or a small batch-size typically ends in well-generalizing, flat regions of the weight space, as indicated by small eigenvalues of the Hessian of the training loss. [...] Key Result In summary, our analysis of the dynamics of SGD in the subspace of the sharpest directions shows that they influence the regions that SGD steers to (where larger learning rate or smaller batch size result in wider regions visited), the…Expand Abstract

View PDF on arXiv

Save to Library

Create Alert

Cite

Launch Research Feed