# An Adaptive Stochastic Sequential Quadratic Programming with Differentiable Exact Augmented Lagrangians

@article{Na2021AnAS,
author={Sen Na and Mihai Anitescu and Mladen Kolar},
journal={ArXiv},
year={2021},
volume={abs/2102.05320}
}
• Published 10 February 2021
• Computer Science
• ArXiv
We consider solving nonlinear optimization problems with stochastic objective and deterministic equality constraints. We assume for the objective that its evaluation, gradient, and Hessian are inaccessible, while one can compute their stochastic estimates by, for example, subsampling. We propose a stochastic algorithm based on sequential quadratic programming (SQP) that uses a differentiable exact augmented Lagrangian as the merit function. To motivate our algorithm design, we first revisit and…

## Figures and Tables from this paper

• Computer Science
ArXiv
• 2021
This work proposes an active-set stochastic sequential quadratic programming algorithm that uses a differentiable exact augmented Lagrangian as the merit function, and adaptively selects the penalty parameters of the augmentedlagrangian, and performs Stochastic line search to decide the stepsize.
• Computer Science, Mathematics
Mathematical Programming
• 2022
There exists a universal weighted averaging scheme that transitions to local convergence at an optimal stage, and still exhibits a superlinear convergence rate nearly (up to a logarithmic factor) matching that of uniform Hessian averaging.
• Computer Science, Mathematics
• 2021
An algorithm that allows inexact subproblem solutions to be employed, which is particularly useful in large-scale settings when the matrices defining the subproblems are too large to form and/or factorize is proposed.
• Computer Science
• 2021
The proposed sequential quadratic optimization algorithm both allows the use of stochastic objective gradient estimates and possesses convergence guarantees even in the setting in which the constraint Jacobians may be rank deficient.
• Computer Science
• 2021
We propose a fast temporal decomposition procedure for solving long-horizon nonlinear dynamic programs. The core of the procedure is sequential quadratic programming (SQP), with a differentiable
• Computer Science
• 2022
A practical adaptive inexact stochastic sequential quadratic programming (PAIS-SQP) method is described and criteria for controlling the sample size and the accuracy in the solutions of the SQP subproblems based on the variance estimates obtained as the optimization progresses is proposed.
• Computer Science, Mathematics
• 2022
The global almost sure convergence guarantee for TR-StoSQP is established, and its empirical performance on both a subset of problems in the CUTEst test set and constrained logistic regression problems using data from the LIBSVM collection is illustrated.
• Computer Science, Mathematics
• 2023
A step-search sequential quadratic programming method is proposed for solving nonlinear equality constrained stochastic optimization problems and a high-probability bound on the iteration complexity of the algorithm to approximate ﬁrst-order stationarity is derived.
• Computer Science, Mathematics
• 2022
Under reasonable assumptions, it is proved that a measure of ﬁrst-order stationarity evaluated at the iterates generated by the proposed algorithm converges to zero in expectation from arbitrary starting points, for both constant and adaptive step size strategies.
• Computer Science
• 2021
The overall complexity bound, which accounts for the adaptivity of the merit parameter sequence, shows that a result comparable to the unconstrained setting (with additional logarithmic factors) holds with high probability.

## References

SHOWING 1-10 OF 141 REFERENCES

• Computer Science
SIAM J. Optim.
• 2021
Under reasonable assumptions, convergence (resp.,~convergence in expectation) from remote starting points is proved for the proposed deterministic (resp,~stochastic) algorithm.
• Computer Science, Mathematics
SIAM J. Optim.
• 2017
The proposed framework extends the classic quasi-Newton methods working in deterministic settings to stochastic settings, and it is proved its almost sure convergence to stationary points.
• Computer Science
SIAM J. Optim.
• 2002
An SQP algorithm that uses a smooth augmented Lagrangian merit function and makes explicit provision for infeasibility in the original problem and the QP subproblems is discussed and a reduced-Hessian semidefinite QP solver (SQOPT) is discussed.
• Mathematics, Computer Science
SIAM J. Optim.
• 2018
It is shown that the inner product test improves upon the well known norm test, and can be used as a basis for an algorithm that is globally convergent on nonconvex functions and enjoys a global linear rate of convergence on strongly convex functions.
• Computer Science
NeurIPS
• 2019
This work proposes to use line-search techniques to automatically set the step-size when training models that can interpolate the data, and proves that SGD with a stochastic variant of the classic Armijo line- search attains the deterministic convergence rates for both convex and strongly-convex functions.
• Computer Science, Mathematics
NeurIPS
• 2018
The proposed algorithm efficiently escapes saddle points and finds approximate local minima for general smooth, nonconvex functions in only $\mathcal{\tilde{O}}(\epsilon^{-3.5})$ stochastic gradient and stochastically Hessian-vector product evaluations.
• Computer Science, Mathematics
SIAM J. Optim.
• 2014
An approach for nonlinear programming based on the direct minimization of an exact differentiable penalty function using trust-region Newton techniques that provides desirable features required for scalability and presents features that are desirable for parametric optimization problems that must be solved in a latency-limited environment.
In this thesis first and second order algorithms are proposed for solving equality constrained finite dimensional minimization problems and optimal control problems with terminal equality constraints using the exact penalty function approach.
• Computer Science, Mathematics
AISTATS
• 2019
This paper theoretically analyzes in the convex and non-convex settings a generalized version of the AdaGrad stepsizes, and shows sufficient conditions for these stepsizes to achieve almost sure asymptotic convergence of the gradients to zero, proving the first guarantee for generalized AdaGrad Stepsizes in the non- Convex setting.
An algorithm for nonlinear programming problems with equality constraints is presented which is globally and superlinearly convergent and incorporates an automatic adjustment rule for the selection of the penalty parameter and avoids the need to evaluate second-order derivatives of the problem functions.