An overview of gradient descent optimization algorithms

@article{Ruder2016AnOO,
  title={An overview of gradient descent optimization algorithms},
  author={Sebastian Ruder},
  journal={CoRR},
  year={2016},
  volume={abs/1609.04747}
}
Gradient descent optimization algorithms, while increasingly popular, are often used as black-box optimizers, as practical explanations of their strengths and weaknesses are hard to come by. This article aims to provide the reader with intuitions with regard to the behaviour of different algorithms that will allow her to put them to use. In the course of this overview, we look at different variants of gradient descent, summarize challenges, introduce the most common optimization algorithms… CONTINUE READING
Highly Influential
This paper has highly influenced a number of papers. REVIEW HIGHLY INFLUENTIAL CITATIONS
Highly Cited
This paper has 199 citations. REVIEW CITATIONS

4 Figures & Tables

Topics

Statistics

05010015020142015201620172018
Citations per Year

200 Citations

Semantic Scholar estimates that this publication has 200 citations based on the available data.

See our FAQ for additional information.

  • GitHub repos referencing this paper

    • dl-labs

      Lab and projects for the Deep Learning course

  • Presentations referencing similar topics