Nesterov's accelerated gradient and momentum as approximations to regularised update descent

Abstract

We present a unifying framework for adapting the update direction in gradient-based iterative optimization methods. As natural special cases we re-derive classical momentum and Nesterov's accelerated gradient method, lending a new intuitive interpretation to the latter algorithm. We show that a new algorithm, which we term Regularised Gradient Descent, can converge more quickly than either Nesterov's algorithm or the classical momentum algorithm.

DOI: 10.1109/IJCNN.2017.7966082

Extracted Key Phrases

3 Figures and Tables

Cite this paper

@article{Botev2017NesterovsAG, title={Nesterov's accelerated gradient and momentum as approximations to regularised update descent}, author={Aleksandar Botev and Guy Lever and David Barber}, journal={2017 International Joint Conference on Neural Networks (IJCNN)}, year={2017}, pages={1899-1903} }