Corpus ID: 166228440

ODE Analysis of Stochastic Gradient Methods with Optimism and Anchoring for Minimax Problems and GANs

  title={ODE Analysis of Stochastic Gradient Methods with Optimism and Anchoring for Minimax Problems and GANs},
  author={E. K. Ryu and Kun Yuan and Wotao Yin},
  • E. K. Ryu, Kun Yuan, Wotao Yin
  • Published 2019
  • Computer Science, Mathematics
  • ArXiv
  • Despite remarkable empirical success, the training dynamics of generative adversarial networks (GAN), which involves solving a minimax game using stochastic gradients, is still poorly understood. In this work, we analyze last-iterate convergence of simultaneous gradient descent (simGD) and its variants under the assumption of convex-concavity, guided by a continuous-time analysis with differential equations. First, we show that simGD, as is, converges with stochastic sub-gradients under strict… CONTINUE READING
    8 Citations

    Figures, Tables, and Topics from this paper.

    Explore Aggressively, Update Conservatively: Stochastic Extragradient Methods with Variable Stepsize Scaling
    • 4
    • Highly Influenced
    • PDF
    LEAD: Least-Action Dynamics for Min-Max Optimization
    Convergence Rates for Boundedly Regular Systems
    • 1
    • PDF
    A Forward-Backward Splitting Method for Monotone Inclusions Without Cocoercivity
    • 34
    • PDF
    Backward-Forward-Reflected-Backward Splitting for Three Operator Monotone Inclusions
    • 6
    • PDF


    Mirror descent in saddle-point problems: Going the extra (gradient) mile
    • 94
    • PDF
    Global Convergence to the Equilibrium of GANs using Variational Inequalities
    • 25
    • PDF
    Gradient descent GAN optimization is locally stable
    • 198
    • PDF
    Reducing Noise in GAN Training with Variance Reduced Extragradient
    • 39
    • PDF
    GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
    • 1,910
    • PDF
    Interaction Matters: A Note on Non-asymptotic Local Convergence of Generative Adversarial Networks
    • 78
    • PDF
    Negative Momentum for Improved Game Dynamics
    • 78
    • PDF
    Stabilizing Adversarial Nets With Prediction Methods
    • 50
    • PDF
    Training GANs with Optimism
    • 188
    • Highly Influential
    • PDF
    The Limit Points of (Optimistic) Gradient Descent in Min-Max Optimization
    • 86
    • PDF