Corpus ID: 214612494

# A new regret analysis for Adam-type algorithms

@article{Alacaoglu2020ANR,
title={A new regret analysis for Adam-type algorithms},
author={Ahmet Alacaoglu and Yura Malitsky and Panayotis Mertikopoulos and Volkan Cevher},
journal={ArXiv},
year={2020},
volume={abs/2003.09729}
}
• Ahmet Alacaoglu, +1 author Volkan Cevher
• Published 2020
• Mathematics, Computer Science
• ArXiv
• In this paper, we focus on a theory-practice gap for Adam and its variants (AMSgrad, AdamNC, etc.). In practice, these algorithms are used with a constant first-order moment parameter $\beta_{1}$ (typically between $0.9$ and $0.99$). In theory, regret guarantees for online convex optimization require a rapidly decaying $\beta_{1}\to0$ schedule. We show that this is an artifact of the standard analysis and propose a novel framework that allows us to derive optimal, data-dependent regret bounds… CONTINUE READING

## Convergence of adaptive algorithms for weakly convex constrained optimization

• Mathematics, Computer Science
• ArXiv
• 2020

#### References

##### Publications referenced by this paper.
SHOWING 1-10 OF 18 REFERENCES

## On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization

• Computer Science, Mathematics
• ICLR
• 2019
VIEW 19 EXCERPTS
HIGHLY INFLUENTIAL

• Computer Science, Mathematics
• ICLR
• 2020
VIEW 14 EXCERPTS
HIGHLY INFLUENTIAL

## On the Convergence of Adam and Beyond

• Computer Science, Mathematics
• ICLR
• 2018
VIEW 37 EXCERPTS
HIGHLY INFLUENTIAL

## Convergence Analyses of Online ADAM Algorithm in Convex Setting and Two-Layer ReLU Neural Network

• Computer Science, Mathematics
• ArXiv
• 2019

## Adam: A Method for Stochastic Optimization

• Computer Science, Mathematics
• ICLR
• 2015
VIEW 6 EXCERPTS
HIGHLY INFLUENTIAL

## A Sufficient Condition for Convergences of Adam and RMSProp

• Fangyu Zou, Wei Liu
• Computer Science, Mathematics
• 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 2019
VIEW 2 EXCERPTS

## On the Convergence of AdaBound and its Connection to SGD

VIEW 1 EXCERPT

VIEW 19 EXCERPTS
HIGHLY INFLUENTIAL

## Adaptive and Self-Confident On-Line Learning Algorithms

• Mathematics, Computer Science
• J. Comput. Syst. Sci.
• 2000

## Logarithmic regret algorithms for online convex optimization

• Computer Science, Mathematics
• Machine Learning
• 2007
VIEW 3 EXCERPTS
HIGHLY INFLUENTIAL