Hybrid Conditional Gradient - Smoothing Algorithms with Applications to Sparse and Low Rank Regularization

@article{Argyriou2014HybridCG,
  title={Hybrid Conditional Gradient - Smoothing Algorithms with Applications to Sparse and Low Rank Regularization},
  author={Andreas Argyriou and Marco Signoretto and Johan A. K. Suykens},
  journal={ArXiv},
  year={2014},
  volume={abs/1404.3591}
}
We study a hybrid conditional gradient - smoothing algorithm (HCGS) for solving composite convex optimization problems which contain several terms over a bounded set. Examples of these include regularization problems with several norms as penalties and a norm constraint. HCGS extends conditional gradient methods to cases with multiple nonsmooth terms, in which standard conditional gradient methods may be difficult to apply. The HCGS algorithm borrows techniques from smoothing proximal methods… 
Model Based Conditional Gradient Method with Armijo-like Line Search
The Conditional Gradient Method is generalized to a class of non-smooth non-convex optimization problems with many applications in machine learning. The proposed algorithm iterates by minimizing
Model Function Based Conditional Gradient Method with Armijo-like Line Search
TLDR
The proposed algorithm iterates by minimizing so-called model functions over the constraint set and is shown to perform favorably on a sparse non-linear robust regression problem and the flexibility of the proposed framework in several matrix factorization formulations is discussed.
Fast column generation for atomic norm regularization
TLDR
The fully corrective Frank-Wolfe (FCFW) algorithm can be made particularly efficient for difficult problems in this family by solving the simplicial or conical subproblems produced by FCFW using a special instance of a classical active set algorithm for quadratic programming.
Solving Separable Nonsmooth Problems Using Frank-Wolfe with Uniform Affine Approximations
TLDR
This paper proposes a modified FW algorithm amenable to nonsmooth functions, subject to a separability assumption, by optimizing for approximation quality over all affine functions, given a neighborhood of interest.
New results on subgradient methods for strongly convex optimization problems with a unified analysis
  • Masaru Ito
  • Mathematics, Computer Science
    Comput. Optim. Appl.
  • 2016
TLDR
A unifying framework for subgradient methods which yields two kinds of methods, namely, the proximal gradient method (PGM) and the conditional gradient method(CGM), unifying several existing methods is proposed.
Decomposition Techniques for Bilinear Saddle Point Problems and Variational Inequalities with Affine Monotone Operators
TLDR
The goal of this paper was to develop alternative decomposition techniques based on LMO for bilinear saddle point problems and for variational inequalities with affine monotone operators.
Adaptive smoothing algorithms for nonsmooth composite convex minimization
  • Q. Tran-Dinh
  • Mathematics, Computer Science
    Comput. Optim. Appl.
  • 2017
TLDR
This work develops a new adaptive smoothing algorithm that has the O1ε-worst-case iteration-complexity while preserves the same complexity-per-iteration as in Nesterov’s method and allows one to automatically update the smoothness parameter at each iteration.
Low-Rank Spectral Optimization via Gauge Duality
TLDR
An algorithm is described that is based on solving a certain constrained eigenvalue optimization problem that corresponds to the gauge dual which, unlike the more typical Lagrange dual, has an especially simple constraint.
A Deterministic Nonsmooth Frank Wolfe Algorithm with Coreset Guarantees
TLDR
A new Frank-Wolfe (FW) type algorithm that is applicable to minimization problems with a nonsmooth convex objective is presented and it is shown that the scheme yields so-called coreset results for various Machine Learning problems including 1-median, Balanced Development, Sparse PCA, Graph Cuts, and the SVM.
Faster Projection-free Online Learning
TLDR
This paper gives an efficient projection-free algorithm that guarantees regret for general online convex optimization with smooth cost functions and one linear optimization computation per iteration.
...
1
2
...

References

SHOWING 1-10 OF 61 REFERENCES
Mirror Prox algorithm for multi-term composite minimization and semi-separable problems
In the paper, we develop a composite version of Mirror Prox algorithm for solving convex–concave saddle point problems and monotone variational inequalities of special structure, allowing to cover
Revisiting Frank-Wolfe: Projection-Free Sparse Convex Optimization
TLDR
A new general framework for convex optimization over matrix factorizations, where every Frank-Wolfe iteration will consist of a low-rank update, is presented, and the broad application areas of this approach are discussed.
PRISMA: PRoximal Iterative SMoothing Algorithm
TLDR
A time variant smoothing strategy is used that allows us to obtain a guarantee that does not depend on knowing in advance the total number of iterations nor a bound on the domain.
Mirror Prox Algorithm for Multi-Term Composite Minimization and Alternating Directions
In the paper, we develop a composite version of Mirror Prox algorithm for solving convex-concave saddle point problems and monotone variational inequalities of special structure, allowing to cover
Convex Optimization without Projection Steps
TLDR
This framework generalizes the sparse greedy algorithm of Frank & Wolfe and its primal-dual analysis by Clarkson 2010 (and the low-rank SDP approach by Hazan 2008) to arbitrary convex domains and gives a convergence proof guaranteeing {\epsilon}-small duality gap after O(1/{\ep silon}) iterations.
A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems
TLDR
A new fast iterative shrinkage-thresholding algorithm (FISTA) which preserves the computational simplicity of ISTA but with a global rate of convergence which is proven to be significantly better, both theoretically and practically.
Duality Between Subgradient and Conditional Gradient Methods
  • F. Bach
  • Computer Science, Mathematics
    SIAM J. Optim.
  • 2015
TLDR
This paper shows the equivalence between mirror descent algorithms and algorithms generalizing the conditional gradient method through convex duality, and implies that for certain problems, such as for supervised machine learning problems with non-smooth losses or problems regularized by non-Smooth regularizers, the primal subgradient method and the dual conditionalgradient method are formally equivalent.
Greedy Algorithms for Structurally Constrained High Dimensional Problems
TLDR
This framework not only unifies existing greedy algorithms by recovering them as special cases but also yields novel ones that solve convex optimization problems that arise when dealing with structurally constrained high-dimensional problems.
The Complexity of Large-scale Convex Programming under a Linear Optimization Oracle
This paper considers a general class of iterative optimization algorithms, referred to as linear-optimization-based convex programming (LCP) methods, for solving large-scale convex programming (CP)
Smoothing and First Order Methods: A Unified Framework
TLDR
It is proved that independently of the structure of the convex nonsmooth function involved, and of the given fast first order iterative scheme, it is always possible to improve the complexity rate and reach an O(\varepsilon^{-1}) efficiency estimate by solving an adequately smoothed approximation counterpart.
...
1
2
3
4
5
...