On the Use of Stochastic Hessian Information in Optimization Methods for Machine Learning

  title={On the Use of Stochastic Hessian Information in Optimization Methods for Machine Learning},
  author={Richard H. Byrd and Gillian M. Chin and Will Neveitt and Jorge Nocedal},
  journal={SIAM Journal on Optimization},
This paper describes how to incorporate sampled curvature information in a NewtonCG method and in a limited memory quasi-Newton method for statistical learning. The motivation for this work stems from supervised machine learning applications involving a very large number of training points. We follow a batch approach, also known in the stochastic optimization literature as a sample average approximation (SAA) approach. Curvature information is incorporated in two sub-sampled Hessian algorithms… CONTINUE READING


Publications citing this paper.
Showing 1-10 of 86 extracted citations

Block-diagonal Hessian-free Optimization

View 7 Excerpts
Highly Influenced

Revisiting Sub-sampled Newton Methods

View 9 Excerpts
Highly Influenced

Training Deep and Recurrent Networks with Hessian-Free Optimization

Neural Networks: Tricks of the Trade • 2012
View 5 Excerpts
Highly Influenced


Publications referenced by this paper.
Showing 1-10 of 17 references

Trust Region Newton Method for Logistic Regression

Journal of Machine Learning Research • 2008
View 1 Excerpt

A Stochastic Approximation Method

Herbert Robbins, Sutton Monro
View 1 Excerpt

MONRO , A stochastic approximation method

Numerical Optimization • 2006

Numerical Optimization. Springer Series in Operations Research

J. Nocedal, S. J. Wright
View 2 Excerpts

Stochastic Programming, Handbook in Operations Research and Management Science

A. Ruszczynski, A. Shapiro
Elsevier Science, • 2003
View 1 Excerpt

Similar Papers

Loading similar papers…