# Newton-type methods for non-convex optimization under inexact Hessian information

@article{Xu2017NewtontypeMF,
title={Newton-type methods for non-convex optimization under inexact Hessian information},
author={Peng Xu and Farbod Roosta-Khorasani and Michael W. Mahoney},
journal={Mathematical Programming},
year={2017},
pages={1-36}
}
• Published 23 August 2017
• Computer Science, Mathematics
• Mathematical Programming
We consider variants of trust-region and adaptive cubic regularization methods for non-convex optimization, in which the Hessian matrix is approximated. Under certain condition on the inexact Hessian, and using approximate solution of the corresponding sub-problems, we provide iteration complexity to achieve $$\varepsilon$$ε-approximate second-order optimality which have been shown to be tight. Our Hessian approximation condition offers a range of advantages as compared with the prior works…
151 Citations
• Computer Science
Comput. Optim. Appl.
• 2020
This work proposes a new trust-region method which employs suitable approximations of the objective function, gradient and Hessian built via random subsampling techniques and shows that the new procedure is more efficient, in terms of overall computational cost, than the standard trust- region scheme with subsampled Hessians.
• Computer Science, Mathematics
SIAM J. Optim.
• 2021
This work draws from matrix perturbation theory to estimate the distance between the subspaces underlying the exact and approximate Hessian matrices in Newton-MR, which extends the application range of the classical Newton-CG beyond convexity to invex problems.
• Computer Science
ArXiv
• 2018
This paper studies a family of stochastic trust region and cubic regularization methods when gradient, Hessian and function values are computed inexactly, and shows the iteration complexity to achieve $\epsilon$-approximate second-order optimality is in the same order with previous work for which gradient and functionvalues are computed exactly.
• Computer Science
• 2018
This paper proposes to compute an approximated Hessian matrix by either uniform or non-uniformly sub-sampling the components of the objective, and develops accelerated adaptive cubic regularization approaches that provide theoretical guarantees on global iteration complexity of O(\epsilon^{-1/3}) with high probability.
• Computer Science, Mathematics
• 2018
This paper proposes to compute an approximated Hessian matrix by either uniformly or non-uniformly sub-sampling the components of the objective and develops accelerated adaptive cubic regularization approaches, which provide theoretical guarantees on global iteration complexity of O (cid:15) − 1 / 3 ) with high probability.
• Computer Science
IMA Journal of Numerical Analysis
• 2022
This approach is a first attempt to introduce inexact Hessian and/or gradient information into the Newton-CG algorithm of Royer & Wright, and derives iteration complexity bounds for achieving $\epsilon$-approximate second-order optimality that match best-known lower bounds.
• Mathematics, Computer Science
SIAM Rev.
• 2020
When the authors use Krylov subspace solutions to approximate the cubic-regularized Newton step, the results recover the strongest known convergence guarantees to approximate second-order stationary points of general smooth nonconvex functions.
• Computer Science
Optimization
• 2020
An extended version of the adaptive cubic regularization method with dynamic inexact Hessian information for nonconvex optimization inherits the innovative use of adaptive accuracy requirements for Hessian approximations introduced in the just quoted paper and additionally employs inexact computations of the gradient.
• Computer Science, Mathematics
Journal of the Operations Research Society of China
• 2022
A new reformulation of the cubic regularization subproblem is proposed, an unconstrained convex problem that requires computing the minimum eigenvalue of the Hessian and a variant of adaptive regularization with cubics (ARC) is derived.
• Computer Science
Proceedings of the International Congress of Mathematicians (ICM 2018)
• 2019
A new general class of inexact second-order algorithms for unconstrained optimization that includes regularization and trust-region variations of Newton's method as well as of their linesearch variants is considered, implying that these methods have optimal worst-case evaluation complexity within a wider class of second- order methods, and that Newton'smethod is suboptimal.

## References

SHOWING 1-10 OF 173 REFERENCES

• Mathematics, Computer Science
• 2011
A general class of second-order iterations for unconstrained optimization that includes regularization and trust-region variants of Newton’s method is considered, implying cubic regularization has optimal worst-case evaluation complexity within this class ofsecond-order methods.
It is shown in this paper that an approximate solution of the trust region problem may be found by the preconditioned conjugate gradient method, and it is shown that the method has the same convergence properties as existing methods based on the dogleg strategy using an approximate Hessian.
• Computer Science, Mathematics
SIAM J. Optim.
• 2017
A randomized second-order method for optimization known as the Newton Sketch, based on performing an approximate Newton step using a randomly projected or sub-sampled Hessian, is proposed, which has super-linear convergence with exponentially high probability and convergence and complexity guarantees that are independent of condition numbers and related problem-dependent quantities.
• Computer Science
• 2016
This paper analyzes an inexact Newton method that solves linear systems approximately using the conjugate gradient (CG) method, and that samples the Hessian and not the gradient (the gradient is assumed to be exact).
• Computer Science
Math. Program. Comput.
• 2010
Methods that obtain the solution of a sequence of parametrized linear systems by factorization are used, and enhancements using high-order polynomial approximation and inverse iteration ensure that the resulting method is both globally and asymptotically at least superlinearly convergent in all cases.
• Computer Science
• 2008
A recursive trust-region method is introduced for the solution of bound-cons-trained nonlinear nonconvex optimization problems for which a hierarchy of descriptions exists, which uses the infinity norm to define the shape of the trust region.
• Mathematics
SIAM J. Optim.
• 2009
This work proposes an extension of the Steihaug-Toint method that allows a solution to be calculated to any prescribed accuracy and includes a parameter that allows the user to take advantage of the tradeoff between the overall number of function evaluations and matrix-vector products associated with the underlying trust-region method.
• Computer Science
SIAM J. Optim.
• 2009
A method is proposed that allows the trust-region norm to be defined independently of the preconditioner over a sequence of evolving low-dimensional subspaces and shows that the method can require significantly fewer function evaluations than other methods.
• Computer Science, Mathematics
• 2016
A variant of a traditional trust region method which is aimed at stochastic optimization and provides a bound on the expected number of iterations the Stochastic algorithm requires to reach accuracy for any $\epsilon>0$.
An algorithm for solving smooth nonconvex optimization problems is proposed that, in the worst-case, takes O(ε−3/2) iterations to drive the norm of the gradient of the objective function below a