# Cycles in adversarial regularized learning

@inproceedings{Mertikopoulos2018CyclesIA,
title={Cycles in adversarial regularized learning},
author={P. Mertikopoulos and Christos H. Papadimitriou and Georgios Piliouras},
booktitle={SODA},
year={2018}
}
• Published in SODA 8 September 2017
• Computer Science
Regularized learning is a fundamental technique in online optimization, machine learning and many other fields of computer science. A natural question that arises in these settings is how regularized learning algorithms behave when faced against each other. We study a natural formulation of this problem by coupling regularized learning dynamics in zero-sum games. We show that the system's behavior is Poincare recurrent, implying that almost every trajectory revisits any (arbitrarily small…
180 Citations

## Figures from this paper

Learning in Matrix Games can be Arbitrarily Complex
• Computer Science
COLT
• 2021
It is proved that replicator dynamics, the continuous-time analogue of Multiplicative Weights Update, even when applied in a very restricted class of games—known as finite matrix games—is rich enough to be able to approximate arbitrary dynamical systems.
No-Regret Learning in Games is Turing Complete
• Computer Science
ArXiv
• 2022
This paper proves Turing completeness of the replicator dynamic on matrix games, one of the simplest possible settings, and implies the undecicability of reachability problems for learning algorithms in games, a special case of which is determining equilibrium convergence.
A novel follow the leader training algorithm for zeros-sum architectures that guarantees convergence to mixed Nash equilibrium without cyclic behaviors and is also efficient for learning in environments with various factors of variations is proposed.
The Mechanics of n-Player Differentiable Games
• Computer Science
ICML
• 2018
The key result is to decompose the second-order dynamics into two components, related to potential games, which reduce to gradient descent on an implicit function; the second relates to Hamiltonian games, a new class of games that obey a conservation law, akin to conservation laws in classical mechanical systems.
On Gradient-Based Learning in Continuous Games
• Computer Science, Economics
SIAM J. Math. Data Sci.
• 2020
A general framework for competitive gradient-based learning is introduced that allows for a wide breadth of learning algorithms including policy gradient reinforcement learning, gradient based bandits, and certain online convex optimization algorithms to be analyzed.
Differentiable Game Mechanics
• Computer Science
J. Mach. Learn. Res.
• 2019
New tools to understand and control the dynamics in n-player differentiable games are developed and basic experiments show SGA is competitive with recently proposed algorithms for finding stable fixed points in GANs -- while at the same time being applicable to, and having guarantees in, much more general cases.
Neural Replicator Dynamics
• Computer Science
ArXiv
• 2019
An elegant one-line change to policy gradient methods is derived that simply bypasses the gradient step through the softmax, yielding a new algorithm titled Neural Replicator Dynamics (NeuRD), which quickly adapts to nonstationarities and outperforms policy gradient significantly in both tabular and function approximation settings.
Mirror descent in saddle-point problems: Going the extra (gradient) mile
• Computer Science
ICLR
• 2019
This work analyzes the behavior of mirror descent in a class of non-monotone problems whose solutions coincide with those of a naturally associated variational inequality-a property which it is called coherence, and shows that optimistic mirror descent (OMD) converges in all coherent problems.
Neural Replicator Dynamics: Multiagent Learning via Hedging Policy Gradients
• Computer Science
AAMAS
• 2020
An elegant one-line change to policy gradient methods is derived that simply bypasses the gradient step through the softmax, yielding a new algorithm titled Neural Replicator Dynamics (NeuRD), which quickly adapts to nonstationarities and outperforms policy gradient significantly in both tabular and function approximation settings.
Poincaré-Bendixson Limit Sets in Multi-Agent Learning
• Economics
AAMAS
• 2022
This paper shows that behaviors consistent with the PoincaréBendixson theorem (limit cycles, but no chaotic attractor) follows purely based on the topological structure of the interaction graph, even for high-dimensional settings with arbitrary number of players and arbitrary payoff matrices.

## References

SHOWING 1-10 OF 54 REFERENCES
Optimization, Learning, and Games with Predictable Sequences
• Computer Science
NIPS
• 2013
It is proved that a version of Optimistic Mirror Descent can be used by two strongly-uncoupled players in a finite zero-sum matrix game to converge to the minimax equilibrium at the rate of O((log T)/T).
Learning in Games via Reinforcement and Regularization
• Economics
Math. Oper. Res.
• 2016
This paper extends several properties of exponential learning, including the elimination of dominated strategies, the asymptotic stability of strict Nash equilibria, and the convergence of time-averaged trajectories in zero-sum games with an interior Nash equilibrium.
Deep Learning Games
• Computer Science
NIPS
• 2016
This work demonstrates an equivalence between global minimizers of the training problem and Nash equilibria in a simple game, and shows how the game can be extended to general acyclic neural networks with differentiable convex gates, establishing a bijection between the Nash equilibrium and critical points of the deep learning problem.
Fast Convergence of Regularized Learning in Games
• Computer Science
NIPS
• 2015
We show that natural classes of regularized learning algorithms with a form of recency bias achieve faster convergence rates to approximate efficiency and to coarse correlated equilibria in
A continuous-time approach to online optimization
• Computer Science
Journal of Dynamics &amp; Games
• 2017
A general class of infinite horizon learning strategies that guarantee an $\mathcal{O}(n^{-1/2})$ regret bound without having to resort to a doubling trick are obtained.
Learning in Games: Robustness of Fast Convergence
• Computer Science, Economics
NIPS
• 2016
We show that learning algorithms satisfying a $\textit{low approximate regret}$ property experience fast convergence to approximate optimality in a large class of repeated games. Our property, which
• Computer Science
NIPS
• 2014
We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and a
Optimization Despite Chaos: Convex Relaxations to Complex Limit Sets via Poincaré Recurrence
• Computer Science
SODA
• 2014
A novel framework is developed that combines ideas from dynamical systems and game theory to produce topological characterizations of system trajectories, and uses information theoretic conservation laws along with Poincare recurrence theory to argue about tightness and optimality of the relaxation techniques.
Persistent patterns: multi-agent learning beyond equilibrium and utility
• Economics
AAMAS
• 2014
An analytic framework for multi-agent learning that is not connected to convergence to an equilibrium concept nor to payoff guarantees for the agents is proposed, based on the contrast between weakly and strongly persistent properties.
The emergence of rational behavior in the presence of stochastic perturbations
• Economics
• 2010
Irrespective of the perturba- tions' magnitude, it is found that strategies which are dominated (even iteratively) eventually become extinct and that the game's strict Nash equilibria are stochastically asymptotically stable.