Evolutionary game theory and multi-agent reinforcement learning

@article{Tuyls2005EvolutionaryGT,
  title={Evolutionary game theory and multi-agent reinforcement learning},
  author={Karl Tuyls and Ann Now{\'e}},
  journal={The Knowledge Engineering Review},
  year={2005},
  volume={20},
  pages={63 - 90}
}
  • K. Tuyls, A. Nowé
  • Published 1 March 2005
  • Computer Science
  • The Knowledge Engineering Review
In this paper we survey the basics of reinforcement learning and (evolutionary) game theory, applied to the field of multi-agent systems. [] Key Method Next we summarize the most important aspects of evolutionary game theory. Finally, we discuss the state-of-the-art of multi-agent reinforcement learning and the mathematical connection with evolutionary game theory.
An Overview of Multi-Agent Reinforcement Learning from Game Theoretical Perspective
TLDR
This work provides a self-contained assessment of the current state-of-the-art MARL techniques from a game theoretical perspective and expects this work to serve as a stepping stone for both new researchers who are about to enter this fast-growing domain and existing domain experts who want to obtain a panoramic view and identify new directions based on recent advances.
Multi-agent learning dynamics
TLDR
This dissertation describes how methods from the field of evolutionary game theory can be used to gain qualitative and theoretical insight into the dynamics of these learning multi-agent systems, as their results are often hard to predict in advance.
The Evolutionary Dynamics of Independent Learning Agents in Population Games
TLDR
A novel unified framework for characterising population dynamics via a single partial differential equation (Theorem 1) is provided and extensive experimental results validating that Theorem 1 holds for a variety of learning methods and population games are presented.
Reinforcement learning for multi-agent systems
TLDR
A survey of multiagent reinforcement learning is given, starting with a review of the different viewpoints on the learning goal, which is a central issue in the field.
Dynamical systems as a level of cognitive analysis of multi-agent learning
  • W. Barfuss
  • Computer Science
    Neural Comput. Appl.
  • 2022
TLDR
This article proposes to embed the dynamical systems description of multi-agent learning into different abstraction levels of cognitive analysis, and proposes an on-line sample-batch temporal-difference algorithm which is characterized by the combination of applying a memory-batch and separated state-action value estimation.
Experience Weighted Learning in Multiagent Systems
TLDR
A learning algorithm for heterogeneous agents with bounded rationality that integrates reinforcement learning as well as fictitious play to evaluate the historical information and adopt mechanisms in evolutionary game to adapt to uncertainty, referred to as experience weighted learning (EWL) is proposed.
Deterministic limit of temporal difference reinforcement learning for stochastic games
TLDR
This work presents a methodological extension, separating the interaction from the adaptation timescale, to derive the deterministic limit of a general class of reinforcement learning algorithms, called temporal difference learning, which is equipped to function in more realistic multistate environments.
A Gradient-Based Reinforcement Learning Algorithm for Multiple Cooperative Agents
TLDR
The Q-value function is applied to estimate the gradient and derive the probability of maximal reward based on estimated gradient ascent (PMR-EGA) algorithm, which can be naturally extended to optimize cooperative stochastic games.
A Collaborative Multiagent Reinforcement Learning Method Based on Policy Gradient Potential
TLDR
A policy gradient potential (PGP) algorithm is proposed that takes PGP as the source of information for guiding the strategy update, as opposed to the gradient itself, to learn the optimal joint strategy that has a maximal global reward.
...
...

References

SHOWING 1-10 OF 113 REFERENCES
Towards a relation between learning agents and evolutionary dynamics
TLDR
This paper examines whether evolutionary game theory, and more specifically the replicator dynamics, is an adequate theoretical model for the study of the dynamics of reinforcement learning agents in a multi-agent system.
Multiagent Reinforcement Learning in Stochastic Games
TLDR
This work designs a multiagent reinforcement learning method which allows agents to learn Nash equilibrium strategies and shows in both theory and experiments that this algorithm converges.
Extended Replicator Dynamics as a Key to Reinforcement Learning in Multi-agent Systems
TLDR
This paper introduces an extension of the Replicator Dynamics from Evolutionary Game Theory and develops a Reinforcement Learning algorithm that attains a stable Nash equilibrium for all types of games.
The Dynamics of Reinforcement Learning in Cooperative Multiagent Systems
TLDR
This work distinguishes reinforcement learners that are unaware of (or ignore) the presence of other agents from those that explicitly attempt to learn the value of joint actions and the strategies of their counterparts, and proposes alternative optimistic exploration strategies that increase the likelihood of convergence to an optimal equilibrium.
Markov Games as a Framework for Multi-Agent Reinforcement Learning
Reinforcement learning of coordination in cooperative multi-agent systems
TLDR
This investigation of reinforcement learning techniques for the learning of coordination in cooperative multi-agent systems focuses on a novel action selection strategy for Q-learning (Watkins 1989), and demonstrates empirically that this extension causes the agents to converge almost always to the optimal joint action even in these difficult cases.
A selection-mutation model for q-learning in multi-agent systems
TLDR
This work shows how the Replicator Dynamics (RD) can be used as a model for Q-learning in games and reveals an interesting connection between the exploitation-exploration scheme from RL and the selection-mutation mechanisms from evolutionary game theory.
On a Dynamical Analysis of Reinforcement Learning in Games: Emergence of Occam's Razor
TLDR
This paper discusses three learning models whose dynamics are related to the Replicator Dynamics and shows how a classical Reinforcement Learning(RL) technique, i.e. Q-learning relates to the RD, allows to better understand the learning process and it allows to determine how complex a RL model should be.
Evolutionary Game Theory
This text introduces current evolutionary game theory--where ideas from evolutionary biology and rationalistic economics meet--emphasizing the links between static and dynamic approaches and
Evolutionary Games and Equilibrium Selection
Evolutionary game theory is one of the most active and rapidly growing areas of research in economics. Unlike traditional game theory models, which assume that all players are fully rational and have
...
...