# Robust Cooperation in the Prisoner's Dilemma: Program Equilibrium via Provability Logic

@article{Brsz2014RobustCI, title={Robust Cooperation in the Prisoner's Dilemma: Program Equilibrium via Provability Logic}, author={Mih{\'a}ly B{\'a}r{\'a}sz and Paul Francis Christiano and Benja Fallenstein and Marcello Herreshoff and Patrick LaVictoire and Eliezer Yudkowsky}, journal={ArXiv}, year={2014}, volume={abs/1401.5577} }

We consider the one-shot Prisoner's Dilemma between algorithms with read-access to one anothers' source codes, and we use the modal logic of provability to build agents that can achieve mutual cooperation in a manner that is robust, in that cooperation does not require exact equality of the agents' source code, and unexploitable, meaning that such an agent never cooperates when its opponent defects. We construct a general framework for such "modal agents", and study their properties.

## 20 Citations

### Robust program equilibrium

- EconomicsTheory and Decision
- 2018

It is argued that this program is similar to the tit for tat strategy for the iterated prisoner’s dilemma and generalizes this approach of turning strategies for the repeated version of a game into programs for the one-shot version of an game to other two-player games and proves that the resulting programs inherit properties of the underlying strategy.

### Parametric Bounded Löb's Theorem and Robust Cooperation of Bounded Agents

- EconomicsArXiv
- 2016

This paper introduces an effective version of Lob's theorem which is applicable given such bounded resources and has powerful implications for the game theory of bounded agents who are able to write proofs about themselves and one another.

### Cooperative and uncooperative institution designs: Surprises and problems in open-source game theory

- Economics, Computer ScienceArXiv
- 2022

This work demonstrates a series of counterintuitive results on open-source games, which are independent of the programming language in which agents are written, and argues that contemporary game theory remains ill-equipped to study program equilibria.

### Game-Theoretic Models of Moral and Other-Regarding Agents (extended abstract)

- EconomicsTARK
- 2021

This work investigates Kantian equilibria in finite normal form games, a class of non-Nashian, morally motivated courses of action that was recently proposed in the economics literature, and proposes some general, intuitive, computationally tractable, otherregarding equilibrium that interpolates between purely self-regarding and Kantian behavior.

### Game-theoretic Models of Moral and Other-Regarding Agents

- EconomicsArXiv
- 2020

This work investigates Kantian equilibria in finite normal form games, a class of non-Nashian, morally motivated courses of action that was recently proposed in the economics literature, and proposes some general, intuitive, computationally tractable, other-regardingEquilibria related to Kantian Equilibria.

### Cooperative and Competitive Reasoning: From Games to Revolutions

- Economics
- 2018

I develop a game theoretic model where players use two different reasoning processes in strategic situations: cooperative and competitive. Players always consider cooperating at first: if they…

### A theory of bounded inductive rationality

- Philosophy
- 2021

The dominant theories of rational decision making assume what we will call logical omniscience. That is, they assume that when facing a decision problem, an agent can perform all relevant…

### Safe Pareto Improvements for Delegated Game Playing

- EconomicsAAMAS
- 2021

It is proved that the notion of safe Pareto improvements is closely related to a notion of outcome correspondence between games and is also shown that under some specific assumptions about how the representatives play games, finding safe Paringo improvements is NP-complete.

### Open Problems in Cooperative AI

- BusinessArXiv
- 2020

This research integrates ongoing work on multi-agent systems, game theory and social choice, human-machine interaction and alignment, natural-language processing, and the construction of social tools and platforms into Cooperative AI, which is an independent bet about the productivity of specific kinds of conversations that involve these and other areas.

### Problem Class Dominance in Predictive Dilemmas

- Law
- 2014

One decision procedure dominates a given one if it performs well on the entire class of problems the given decision procedure performs well on, and then goes on to perform well on other problems that…

## References

SHOWING 1-10 OF 26 REFERENCES

### Program equilibrium—a program reasoning approach

- Economics, Computer ScienceInt. J. Game Theory
- 2013

This paper investigates an approach to program equilibrium in which richer conditions are allowed, based on model checking—one of the most successful approaches to reasoning about programs, and introduces a decision-tree model of strategies, which may be conditioned on strategies of others.

### Modeling rational players I

- Economics
- 1987

This paper focuses on the non-cooperative analysis of games with some dynamic structure in which the choice of one move or another during the play of the game may convey valuable information to the other players.

### Modeling Rational Players: Part I

- EconomicsEconomics and Philosophy
- 1987

Game theory has proved a useful tool in the study of simple economic models. However, numerous foundational issues remain unresolved. The situation is particularly confusing in respect of the…

### Rationality, Computability, and Nash Equilibrium

- Economics
- 1992

Suppose two agents play a game, each using a computable algorithm to decide what to do, these algorithms being common knowledge. The author shows that it is possible to act rationally provided he…

### The evolution of cooperation.

- BiologyScience
- 1981

A model is developed based on the concept of an evolutionarily stable strategy in the context of the Prisoner's Dilemma game to show how cooperation based on reciprocity can get started in an asocial world, can thrive while interacting with a wide range of other strategies, and can resist invasion once fully established.

### Some notes on Church's thesis and the theory of games

- Economics
- 1990

This paper considers games in normal form played by Turing Machines. The machines are fed as input all the relevent information and then are required to play the game. Some ‘impossibility’ results…

### Definable and Contractible Contracts

- Economics
- 2012

This paper analyzes Bayesian normal form games in which players write contracts that condition their actions on the contracts of the other players. These contracts are required to be representable in…