# Efficient Model-Based Multi-Agent Mean-Field Reinforcement Learning

@article{Pasztor2021EfficientMM, title={Efficient Model-Based Multi-Agent Mean-Field Reinforcement Learning}, author={Barna Pasztor and Ilija Bogunovic and Andreas Krause}, journal={ArXiv}, year={2021}, volume={abs/2107.04050} }

Learning in multi-agent systems is highly challenging due to the inherent complexity introduced by agents’ interactions. We tackle systems with a huge population of interacting agents (e.g., swarms) via Mean-Field Control (MFC). MFC considers an asymptotically infinite population of identical agents that aim to collaboratively maximize the collective reward. Specifically, we consider the case of unknown system dynamics where the goal is to simultaneously optimize for the rewards and learn from…

## 3 Citations

On the Approximation of Cooperative Heterogeneous Multi-Agent Reinforcement Learning (MARL) using Mean Field Control (MFC)

- Computer ScienceArXiv
- 2021

This work considers a collection of Npop heterogeneous agents that can be segregated into K classes such that the k-th class contains Nk homogeneous agents and proves approximation guarantees of the MARL problem for this heterogeneous system by its corresponding MFC problem.

A General Framework for Learning Mean-Field Games

- Computer Science, MathematicsArXiv
- 2020

A general mean-field game framework for simultaneous learning and decision-making in stochastic games with a large population is presented, and a specific GMF-V algorithm based on Q-learning is demonstrated to be efficient and robust in terms of convergence and learning accuracy.

Learning Graphon Mean Field Games and Approximate Nash Equilibria

- Computer Science, MathematicsArXiv
- 2021

This work proposes a novel discrete-time formulation for graphon mean field games as the limit of non-linear dense graph Markov games with weak interaction and successfully applies policy gradient reinforcement learning in conjunction with sequential Monte Carlo methods.

## References

SHOWING 1-10 OF 67 REFERENCES

Mean Field Multi-Agent Reinforcement Learning

- Computer ScienceICML
- 2018

Existing multi-agent reinforcement learning methods are limited typically to a small number of agents. When the agent number increases largely, the learning becomes intractable due to the curse of…

On the Convergence of Model Free Learning in Mean Field Games

- Mathematics, Computer ScienceAAAI
- 2020

This paper analyzes in full generality the convergence of a fictitious iterative scheme using any single agent learning algorithm at each step of the Mean Field MAS, and shows for the first time convergence of model free learning algorithms towards non-stationary MFG equilibria.

Reinforcement Learning in Stationary Mean-field Games

- Computer Science, MathematicsAAMAS
- 2019

This paper studies reinforcement learning in a specific class of multi-agent systems systems called mean-field games, and presents two reinforcement learning algorithms that converge to the right solution under mild technical conditions.

Unified reinforcement Q-learning for mean field game and control problems

- Computer Science, MathematicsMathematics of Control, Signals, and Systems
- 2022

A Reinforcement Learning (RL) algorithm to solve infinite horizon asymptotic Mean Field Game (MFG) and Mean Field Control (MFC) problems is presented, described as a unified two-timescale Mean Field Q-learning.

Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learning

- Computer Science, MathematicsArXiv
- 2019

This work introduces generic model-free algorithms based on the state-action value function at the mean field level and proves convergence for a prototypical Q-learning method for mean field control problems.

Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models

- Computer Science, MathematicsNeurIPS
- 2018

This paper proposes a new algorithm called probabilistic ensembles with trajectory sampling (PETS) that combines uncertainty-aware deep network dynamics models with sampling-based uncertainty propagation, which matches the asymptotic performance of model-free algorithms on several challenging benchmark tasks, while requiring significantly fewer samples.

Reinforcement Learning for Mean Field Game

- Computer Science, MathematicsArXiv
- 2019

A posterior sampling based approach for reinforcement learning in the mean-field game, where each agent samples a transition probability from the previous transitions, which constitute an MFE in an action coupled stochastic game setting in an episodic framework.

Deep Reinforcement Learning for Multiagent Systems: A Review of Challenges, Solutions, and Applications

- Computer Science, MathematicsIEEE Transactions on Cybernetics
- 2020

A survey of different approaches to problems related to multiagent deep RL (MADRL) is presented, including nonstationarity, partial observability, continuous state and action spaces, multiagent training schemes, and multiagent transfer learning.

A Comprehensive Survey of Multiagent Reinforcement Learning

- Computer ScienceIEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)
- 2008

The benefits and challenges of MARL are described along with some of the problem domains where the MARL techniques have been applied, and an outlook for the field is provided.

A General Framework for Learning Mean-Field Games

- Computer Science, MathematicsArXiv
- 2020

A general mean-field game framework for simultaneous learning and decision-making in stochastic games with a large population is presented, and a specific GMF-V algorithm based on Q-learning is demonstrated to be efficient and robust in terms of convergence and learning accuracy.