Corpus ID: 34750601

First-order Methods Almost Always Avoid Saddle Points

@inproceedings{Lee2019FirstorderMA,
  title={First-order Methods Almost Always Avoid Saddle Points},
  author={Jason D. Lee and Ioannis Panageas and Georgios Piliouras and Max Simchowitz and Michael I. Jordan and Benjamin Recht},
  booktitle={NeurIPS},
  year={2019}
}
  • Jason D. Lee, Ioannis Panageas, +3 authors Benjamin Recht
  • Published in NeurIPS 2019
  • Computer Science, Mathematics
  • We establish that first-order methods avoid saddle points for almost all initializations. Our results apply to a wide variety of first-order methods, including gradient descent, block coordinate descent, mirror descent and variants thereof. The connecting thread is that such algorithms can be studied from a dynamical systems perspective in which appropriate instantiations of the Stable Manifold Theorem allow for a global stability analysis. Thus, neither access to second-order derivative… CONTINUE READING

    Figures and Topics from this paper.

    Citations

    Publications citing this paper.
    SHOWING 1-10 OF 77 CITATIONS

    Provable Bregman-divergence based Methods for Nonconvex and Non-Lipschitz Problems

    VIEW 23 EXCERPTS
    CITES METHODS, BACKGROUND & RESULTS
    HIGHLY INFLUENCED

    Alternating Minimizations Converge to Second-Order Optimal Solutions

    VIEW 8 EXCERPTS
    CITES BACKGROUND & METHODS
    HIGHLY INFLUENCED

    Differentiable Game Mechanics

    VIEW 5 EXCERPTS
    CITES BACKGROUND & METHODS
    HIGHLY INFLUENCED

    Fast Algorithms for Sparse Reduced-Rank Regression

    VIEW 5 EXCERPTS
    CITES BACKGROUND & METHODS
    HIGHLY INFLUENCED

    Escaping from saddle points on Riemannian manifolds

    VIEW 2 EXCERPTS
    CITES BACKGROUND
    HIGHLY INFLUENCED

    Escaping from saddle points on Riemannian manifolds

    VIEW 2 EXCERPTS
    CITES BACKGROUND
    HIGHLY INFLUENCED

    Extending the step-size restriction for gradient descent to avoid strict saddle points

    VIEW 5 EXCERPTS
    CITES BACKGROUND
    HIGHLY INFLUENCED

    Multichannel Sparse Blind Deconvolution on the Sphere

    VIEW 4 EXCERPTS
    CITES METHODS
    HIGHLY INFLUENCED

    A general system of differential equations to model first order adaptive algorithms

    VIEW 5 EXCERPTS
    CITES BACKGROUND & RESULTS
    HIGHLY INFLUENCED

    FILTER CITATIONS BY YEAR

    2017
    2020

    CITATION STATISTICS

    • 11 Highly Influenced Citations

    • Averaged 24 Citations per year from 2017 through 2019

    • 72% Increase in citations per year in 2019 over 2018

    References

    Publications referenced by this paper.
    SHOWING 1-10 OF 55 REFERENCES

    A Generic Approach for Escaping Saddle points

    VIEW 1 EXCERPT