# Büchi Objectives in Countable MDPs

@inproceedings{Kiefer2019BchiOI, title={B{\"u}chi Objectives in Countable MDPs}, author={S. Kiefer and R. Mayr and M. Shirmohammadi and P. Totzke}, booktitle={ICALP}, year={2019} }

We study countably infinite Markov decision processes with Buchi objectives, which ask to visit a given subset of states infinitely often. A question left open by T.P. Hill in 1979 is whether there always exist $\varepsilon$-optimal Markov strategies, i.e., strategies that base decisions only on the current state and the number of steps taken so far. We provide a negative answer to this question by constructing a non-trivial counterexample. On the other hand, we show that Markov strategies with… Expand

#### 8 Citations

Büchi Objectives in Countable MDPs

- 2019

We study countably infinite Markov decision processes with Büchi objectives, which ask to visit a given subset F of states infinitely often. A question left open by T.P. Hill in 1979 [10] is whether… Expand

How to Play in Infinite MDPs

- 2020

Markov decision processes (MDPs) are a standard model for dynamic systems that exhibit both stochastic and nondeterministic behavior. For MDPs with finite state space it is known that for a wide… Expand

How to Play in Infinite MDPs

- 2020

Markov decision processes (MDPs) are a standard model for dynamic systems that exhibit both stochastic and nondeterministic behavior. For MDPs with finite state space it is known that for a wide… Expand

Transience in Countable MDPs

- Computer Science, Mathematics
- CONCUR
- 2021

The Transience objective is not to visit any state infinitely often. While this is not possible in any finite Markov Decision Process (MDP), it can be satisfied in countably infinite ones, e.g., if… Expand

Strategy Complexity of Parity Objectives in Countable MDPs

- Computer Science, Mathematics
- CONCUR
- 2020

A complete picture of the exact strategy complexity of $\varepsilon$-optimal strategies (and optimal strategies, where they exist) for all subclasses of parity objectives in the Mostowski hierarchy is provided. Expand

Taming denumerable Markov decision processes with decisiveness

- Computer Science
- ArXiv
- 2020

This paper explores how to extend the notion of decisiveness to Markov decision processes, and whether these notions yield model checking procedures concerning the infimum and supremum probabilities of reachability properties. Expand

Symbolic controller synthesis for Büchi specifications on stochastic systems

- Computer Science, Engineering
- HSCC
- 2020

The original policy synthesis problem for continuous-state controlled Markov processes evolving in discrete time is reduced to a Büchi game under a fairness assumption and upper and lower bounds on winning sets are characterized as nested fixed point expressions in the μ-calculus. Expand

Strategy Complexity of Mean Payoff, Total Payoff and Point Payoff Objectives in Countable MDPs

- Computer Science, Mathematics
- CONCUR
- 2021

A complete picture is established of the strategy complexity of countably infinite Markov decision processes with real-valued transition rewards, i.e., how much memory is necessary and sufficient for ε-optimal (resp. optimal) strategies. Expand

#### References

SHOWING 1-10 OF 23 REFERENCES

Büchi Objectives in Countable MDPs

- 2019

We study countably infinite Markov decision processes with Büchi objectives, which ask to visit a given subset F of states infinitely often. A question left open by T.P. Hill in 1979 [10] is whether… Expand

Parity objectives in countable MDPs

- Mathematics, Computer Science
- 2017 32nd Annual ACM/IEEE Symposium on Logic in Computer Science (LICS)
- 2017

We study countably infinite MDPs with parity objectives, and special cases with a bounded number of colors in the Mostowski hierarchy (including reachability, safety, Büchi and co-Büchi).

On the Existence of Good Markov Strategies

- Mathematics
- 1979

In contrast to the known fact that there are gambling problems based on a finite state space for which no stationary family of strategies is at all good, in every such problem there always exist… Expand

On the existence of stationary optimal strategies

- Mathematics
- 1969

The question with which this paper is concerned is roughly speaking: In a gambling situation or dynamical programming situation, are strategies that take the past into account any better than those… Expand

A survey of stochastic ω-regular games

- Computer Science, Mathematics
- J. Comput. Syst. Sci.
- 2012

We summarize classical and recent results about two-player games played on graphs with @w-regular objectives. These games have applications in the verification and synthesis of reactive systems.… Expand

Quantitative stochastic parity games

- Mathematics, Computer Science
- SODA '04
- 2004

The existence of optimal pure memoryless strategies together with the polynomial-time solution for the one-player case implies that the quantitative two-player stochastic parity game problem is in NP ∩ co-NP, which generalizes a result of Condon for Stochastic games with reachability objectives. Expand

Principles of model checking

- Computer Science
- 2008

Principles of Model Checking offers a comprehensive introduction to model checking that is not only a text suitable for classroom use but also a valuable reference for researchers and practitioners in the field. Expand

A survey of computational complexity results in systems and control

- Computer Science
- Autom.
- 2000

This paper considers problems related to stability or stabilizability of linear systems with parametric uncertainty, robust control, time-varying linear systems, nonlinear and hybrid systems, and stochastic optimal control. Expand

Determinacy and Optimal Strategies in Stochastic Games

- Mathematics
- 2009

Zabývame se determinovanosti stochastických tahových her s uplnou informaci a s výhernimi cili dosažitelnosti, bezpecnosti a Buchiho akceptacni podminky. Odděleně rozebirame situaci pro konecně… Expand

goal problems in gambling theory

- Mathematics
- 2009

A short introduction to goal problems in abstract gambling theory is given, along with statementes of some of the main theorems and a number of examples, open problems and references. Emphasis is on… Expand