Corpus ID: 166228440

ODE Analysis of Stochastic Gradient Methods with Optimism and Anchoring for Minimax Problems and GANs

  title={ODE Analysis of Stochastic Gradient Methods with Optimism and Anchoring for Minimax Problems and GANs},
  author={E. K. Ryu and Kun Yuan and Wotao Yin},
  • E. K. Ryu, Kun Yuan, Wotao Yin
  • Published 2019
  • Computer Science, Mathematics
  • ArXiv
  • Despite remarkable empirical success, the training dynamics of generative adversarial networks (GAN), which involves solving a minimax game using stochastic gradients, is still poorly understood. In this work, we analyze last-iterate convergence of simultaneous gradient descent (simGD) and its variants under the assumption of convex-concavity, guided by a continuous-time analysis with differential equations. First, we show that simGD, as is, converges with stochastic sub-gradients under strict… CONTINUE READING
    9 Citations

    Figures, Tables, and Topics from this paper

    Training GANs with predictive projection centripetal acceleration
    Explore Aggressively, Update Conservatively: Stochastic Extragradient Methods with Variable Stepsize Scaling
    • 7
    • Highly Influenced
    • PDF
    LEAD: Least-Action Dynamics for Min-Max Optimization
    • PDF
    Convergence Rates for Boundedly Regular Systems
    • 1
    • PDF
    A Forward-Backward Splitting Method for Monotone Inclusions Without Cocoercivity
    • 36
    • PDF
    Backward-Forward-Reflected-Backward Splitting for Three Operator Monotone Inclusions
    • 7
    • PDF


    Mirror descent in saddle-point problems: Going the extra (gradient) mile
    • 100
    • PDF
    Global Convergence to the Equilibrium of GANs using Variational Inequalities
    • 26
    • PDF
    Gradient descent GAN optimization is locally stable
    • 208
    • PDF
    Reducing Noise in GAN Training with Variance Reduced Extragradient
    • 41
    • PDF
    Interaction Matters: A Note on Non-asymptotic Local Convergence of Generative Adversarial Networks
    • 82
    • PDF
    GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
    • 2,281
    • PDF
    Negative Momentum for Improved Game Dynamics
    • 86
    • PDF
    Stabilizing Adversarial Nets With Prediction Methods
    • 53
    • PDF
    Training GANs with Optimism
    • 200
    • Highly Influential
    • PDF
    The Limit Points of (Optimistic) Gradient Descent in Min-Max Optimization
    • 95
    • PDF