Cycles in adversarial regularized learning

@inproceedings{Mertikopoulos2018CyclesIA,
  title={Cycles in adversarial regularized learning},
  author={P. Mertikopoulos and Christos H. Papadimitriou and Georgios Piliouras},
  booktitle={SODA},
  year={2018}
}
Regularized learning is a fundamental technique in online optimization, machine learning and many other fields of computer science. A natural question that arises in these settings is how regularized learning algorithms behave when faced against each other. We study a natural formulation of this problem by coupling regularized learning dynamics in zero-sum games. We show that the system's behavior is Poincare recurrent, implying that almost every trajectory revisits any (arbitrarily small… 

Figures from this paper

Learning in Matrix Games can be Arbitrarily Complex
TLDR
It is proved that replicator dynamics, the continuous-time analogue of Multiplicative Weights Update, even when applied in a very restricted class of games—known as finite matrix games—is rich enough to be able to approximate arbitrary dynamical systems.
No-Regret Learning in Games is Turing Complete
TLDR
This paper proves Turing completeness of the replicator dynamic on matrix games, one of the simplest possible settings, and implies the undecicability of reachability problems for learning algorithms in games, a special case of which is determining equilibrium convergence.
Follow the Neurally-Perturbed Leader for Adversarial Training
TLDR
A novel follow the leader training algorithm for zeros-sum architectures that guarantees convergence to mixed Nash equilibrium without cyclic behaviors and is also efficient for learning in environments with various factors of variations is proposed.
The Mechanics of n-Player Differentiable Games
TLDR
The key result is to decompose the second-order dynamics into two components, related to potential games, which reduce to gradient descent on an implicit function; the second relates to Hamiltonian games, a new class of games that obey a conservation law, akin to conservation laws in classical mechanical systems.
On Gradient-Based Learning in Continuous Games
TLDR
A general framework for competitive gradient-based learning is introduced that allows for a wide breadth of learning algorithms including policy gradient reinforcement learning, gradient based bandits, and certain online convex optimization algorithms to be analyzed.
Differentiable Game Mechanics
TLDR
New tools to understand and control the dynamics in n-player differentiable games are developed and basic experiments show SGA is competitive with recently proposed algorithms for finding stable fixed points in GANs -- while at the same time being applicable to, and having guarantees in, much more general cases.
Neural Replicator Dynamics
TLDR
An elegant one-line change to policy gradient methods is derived that simply bypasses the gradient step through the softmax, yielding a new algorithm titled Neural Replicator Dynamics (NeuRD), which quickly adapts to nonstationarities and outperforms policy gradient significantly in both tabular and function approximation settings.
Mirror descent in saddle-point problems: Going the extra (gradient) mile
TLDR
This work analyzes the behavior of mirror descent in a class of non-monotone problems whose solutions coincide with those of a naturally associated variational inequality-a property which it is called coherence, and shows that optimistic mirror descent (OMD) converges in all coherent problems.
Neural Replicator Dynamics: Multiagent Learning via Hedging Policy Gradients
TLDR
An elegant one-line change to policy gradient methods is derived that simply bypasses the gradient step through the softmax, yielding a new algorithm titled Neural Replicator Dynamics (NeuRD), which quickly adapts to nonstationarities and outperforms policy gradient significantly in both tabular and function approximation settings.
Poincaré-Bendixson Limit Sets in Multi-Agent Learning
TLDR
This paper shows that behaviors consistent with the PoincaréBendixson theorem (limit cycles, but no chaotic attractor) follows purely based on the topological structure of the interaction graph, even for high-dimensional settings with arbitrary number of players and arbitrary payoff matrices.
...
...

References

SHOWING 1-10 OF 54 REFERENCES
Optimization, Learning, and Games with Predictable Sequences
TLDR
It is proved that a version of Optimistic Mirror Descent can be used by two strongly-uncoupled players in a finite zero-sum matrix game to converge to the minimax equilibrium at the rate of O((log T)/T).
Learning in Games via Reinforcement and Regularization
TLDR
This paper extends several properties of exponential learning, including the elimination of dominated strategies, the asymptotic stability of strict Nash equilibria, and the convergence of time-averaged trajectories in zero-sum games with an interior Nash equilibrium.
Deep Learning Games
TLDR
This work demonstrates an equivalence between global minimizers of the training problem and Nash equilibria in a simple game, and shows how the game can be extended to general acyclic neural networks with differentiable convex gates, establishing a bijection between the Nash equilibrium and critical points of the deep learning problem.
Fast Convergence of Regularized Learning in Games
We show that natural classes of regularized learning algorithms with a form of recency bias achieve faster convergence rates to approximate efficiency and to coarse correlated equilibria in
A continuous-time approach to online optimization
TLDR
A general class of infinite horizon learning strategies that guarantee an $\mathcal{O}(n^{-1/2})$ regret bound without having to resort to a doubling trick are obtained.
Learning in Games: Robustness of Fast Convergence
We show that learning algorithms satisfying a $\textit{low approximate regret}$ property experience fast convergence to approximate optimality in a large class of repeated games. Our property, which
Generative Adversarial Nets
We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and a
Optimization Despite Chaos: Convex Relaxations to Complex Limit Sets via Poincaré Recurrence
TLDR
A novel framework is developed that combines ideas from dynamical systems and game theory to produce topological characterizations of system trajectories, and uses information theoretic conservation laws along with Poincare recurrence theory to argue about tightness and optimality of the relaxation techniques.
Persistent patterns: multi-agent learning beyond equilibrium and utility
TLDR
An analytic framework for multi-agent learning that is not connected to convergence to an equilibrium concept nor to payoff guarantees for the agents is proposed, based on the contrast between weakly and strongly persistent properties.
The emergence of rational behavior in the presence of stochastic perturbations
TLDR
Irrespective of the perturba- tions' magnitude, it is found that strategies which are dominated (even iteratively) eventually become extinct and that the game's strict Nash equilibria are stochastically asymptotically stable.
...
...