Corpus ID: 10284405

New Insights and Perspectives on the Natural Gradient Method

@article{Martens2020NewIA,
  title={New Insights and Perspectives on the Natural Gradient Method},
  author={J. Martens},
  journal={J. Mach. Learn. Res.},
  year={2020},
  volume={21},
  pages={146:1-146:76}
}

J. Martens

Published 2020

Computer Science, Mathematics

J. Mach. Learn. Res.

Natural gradient descent is an optimization method traditionally motivated from the perspective of information geometry, and works well for many applications as an alternative to stochastic gradient descent. In this paper we critically analyze this method and its properties, and show how it can be viewed as a type of approximate 2nd-order optimization method, where the Fisher information matrix can be viewed as an approximation of the Hessian. This perspective turns out to have significant… CONTINUE READING

View PDF on arXiv

Save to Library

Create Alert

Cite

Launch Research Feed