Corpus ID: 212414659

@article{Dfossez2020OnTC,
author={Alexandre D{\'e}fossez and L. Bottou and Francis R. Bach and Nicolas Usunier},
journal={ArXiv},
year={2020},
volume={abs/2003.02395}
}
• Alexandre Défossez, +1 author Nicolas Usunier
• Published 2020
• Mathematics, Computer Science
• ArXiv
• We provide a simple proof of the convergence of the optimization algorithms Adam and Adagrad with the assumptions of smooth gradients and almost sure uniform bound on the $\ell_\infty$ norm of the gradients. This work builds on the techniques introduced by Ward et al. (2019) and extends them to the Adam optimizer. We show that in expectation, the squared norm of the objective gradient averaged over the trajectory has an upper-bound which is explicit in the constants of the problem, parameters… CONTINUE READING

#### References

##### Publications referenced by this paper.
SHOWING 1-10 OF 19 REFERENCES
Adam: A Method for Stochastic Optimization
• Computer Science, Mathematics
• 2015
• 49,675
• Highly Influential
• PDF
On the Convergence of Adam and Beyond
• Computer Science, Mathematics
• 2018
• 838
• Highly Influential
• PDF
• Computer Science, Mathematics
• 2010
• 6,012
• Highly Influential
• PDF
A Sufficient Condition for Convergences of Adam and RMSProp
• Fangyu Zou, Wei Liu
• Computer Science, Mathematics
• 2019
• 50
• Highly Influential
• PDF