Quadratic optimization with orthogonality constraint: explicit Łojasiewicz exponent and linear convergence of retraction-based line-search and stochastic variance-reduced gradient methods

@article{Liu2019QuadraticOW,
  title={Quadratic optimization with orthogonality constraint: explicit Łojasiewicz exponent and linear convergence of retraction-based line-search and stochastic variance-reduced gradient methods},
  author={Huikang Liu and Anthony Man-Cho So and Weijie Wu},
  journal={Mathematical Programming},
  year={2019},
  pages={1-48}
}
The problem of optimizing a quadratic form over an orthogonality constraint (QP-OC for short) is one of the most fundamental matrix optimization problems and arises in many applications. In this paper, we characterize the growth behavior of the objective function around the critical points of the QP-OC problem and demonstrate how such characterization can be used to obtain strong convergence rate results for iterative methods that exploit the manifold structure of the orthogonality constraint… 
A family of inexact SQA methods for non-smooth convex minimization with provable convergence guarantees based on the Luo–Tseng error bound property
TLDR
This work proves that when the problem possesses the so-called Luo–Tseng error bound (EB) property, IRPN converges globally to an optimal solution, and the local convergence rate of the sequence of iterates generated by IRPN is linear, superlinear, or even quadratic, depending on the choice of parameters of the algorithm.
WEAKLY CONVEX OPTIMIZATION OVER STIEFEL MANIFOLD
TLDR
These are the first convergence guarantees for using Riemannian subgradient-type methods to optimize a class of nonconvex nonsmooth functions over the Stiefel manifold.
Factorization of completely positive matrices using iterative projected gradient steps
TLDR
This work proposes a projected gradient algorithm with parameters that take into account the effects of relaxation and inertia and shows that the whole sequence of generated iterates converges to a critical point of the objective function.
Convergence Analysis of Alternating Projection Method for Nonconvex Sets
TLDR
This paper formalizes two properties of proper, lower semi-continuous and semi-algebraic sets: the three-point property for all possible iterates and the local contraction property that serves as the non-expensiveness property of the projector but only for the iterates that are close enough to each other.
Convergence Analysis of Alternating Nonconvex Projections
TLDR
A new convergence analysis framework is established to show that if one set satisfies the three point property and the other one obeys the local contraction property, the iterates generated by alternating projections is a convergent sequence and converges to a critical point.
Linear Convergence of a Proximal Alternating Minimization Method with Extrapolation for $\ell_1$-Norm Principal Component Analysis
TLDR
This paper proposes a proximal alternating minimization method with extrapolation (PAMe) for solving a two-block reformulation of the L1-PCA problem and shows via numerical experiments that PAMe is competitive with a host of existing methods.
Proximal Gradient Method for Nonsmooth Optimization over the Stiefel Manifold
TLDR
It is proved that the proposed retraction-based proximal gradient method globally converges to a stationary point and Iteration complexity for obtaining an $\epsilon$-stationary solution is analyzed.
Nonsmooth Optimization over Stiefel Manifold: Riemannian Subgradient Methods
TLDR
This paper studies optimization problems over the Stiefel manifold with nonsmooth objective function, and establishes convergence results to a broader class of compact Riemannian manifolds embedded in Euclidean space.
Nonmonotone inexact restoration approach for minimization with orthogonality constraints
TLDR
This paper gives a suitable characterization of the tangent set of the orthogonality constraints, allowing the nonmonotone variation of the inexact restoration method to deal with the minimization phase efficiently and employ the Cayley transform to bring a point in the tangents back to feasibility, leading to a SVD-free restoration phase.
A globally and linearly convergent PGM for zero-norm regularized quadratic optimization with sphere constraint
TLDR
The KL property of exponent 1/2 is established for its extended-valued objective function and a globally and linearly convergent proximal gradient method (PGM) is developed for zero-norm regularized quadratic optimization with a sphere constraint.
...
1
2
3
4
...

References

SHOWING 1-10 OF 65 REFERENCES
Quadratic Optimization with Orthogonality Constraints: Explicit Lojasiewicz Exponent and Linear Convergence of Line-Search Methods
TLDR
This work gives an explicit estimate of the exponent in a Lojasiewicz inequality for the (non-convex) set of critical points of the aforementioned class of problems, and establishes the linear convergence of a large class of line-search methods.
A unified approach to error bounds for structured convex optimization problems
TLDR
A new framework for establishing error bounds for a class of structured convex optimization problems, in which the objective function is the sum of a smooth convex function and a general closed proper convexfunction, is presented.
From error bounds to the complexity of first-order descent methods for convex functions
TLDR
It is shown that error bounds can be used as effective tools for deriving complexity results for first-order descent methods in convex minimization and how KL inequalities can in turn be employed to compute new complexity bounds for a wealth of descent methods for convex problems.
Calculus of the Exponent of Kurdyka–Łojasiewicz Inequality and Its Applications to Linear Convergence of First-Order Methods
TLDR
The Kurdyka–Łojasiewicz exponent is studied, an important quantity for analyzing the convergence rate of first-order methods, and various calculus rules are developed to deduce the KL exponent of new (possibly nonconvex and nonsmooth) functions formed from functions with known KL exponents.
A feasible method for optimization with orthogonality constraints
TLDR
The Cayley transform is applied—a Crank-Nicolson-like update scheme—to preserve the constraints and based on it, curvilinear search algorithms with lower flops are developed with high efficiency for polynomial optimization, nearest correlation matrix estimation and extreme eigenvalue problems.
Convergence Results for Projected Line-Search Methods on Varieties of Low-Rank Matrices Via Łojasiewicz Inequality
TLDR
Convergence results for projected line-search methods on the real-algebraic variety $\mathcal{M}_{\le k}$ of real $m \times n$ matrices of rank at most $k$ are derived.
New fractional error bounds for polynomial systems with applications to Hölderian stability in optimization and spectral theory of tensors
TLDR
New fractional error bounds for polynomial systems with exponents explicitly determined by the dimension of the underlying space and the number/degree of the involved polynomials are derived.
Guaranteed Matrix Completion via Nonconvex Factorization
  • Ruoyu Sun, Z. Luo
  • Computer Science
    2015 IEEE 56th Annual Symposium on Foundations of Computer Science
  • 2015
TLDR
This paper establishes a theoretical guarantee for the factorization based formulation to correctly recover the underlying low-rank matrix, and is the first one that provides exact recovery guarantee for many standard algorithms such as gradient descent, SGD and block coordinate gradient descent.
Convergence of the Iterates of Descent Methods for Analytic Cost Functions
TLDR
It is shown that the iterates of numerical descent algorithms, for an analytic cost function, share this convergence property if they satisfy certain natural descent conditions and strengthen classical "weak convergence" results for descent methods to "strong limit-point convergence" for a large class of cost functions of practical interest.
Moment inequalities for sums of random matrices and their applications in optimization
TLDR
This paper shows that an SDP-based algorithm of Nemirovski, which is developed for solving a class of quadratic optimization problems with orthogonality constraints, has a logarithmic approximation guarantee, and improves upon the polynomial approximation guarantee established earlier by NemiroVSki.
...
1
2
3
4
5
...