# Büchi Objectives in Countable MDPs

@inproceedings{Kiefer2019BchiOI,
title={B{\"u}chi Objectives in Countable MDPs},
author={S. Kiefer and R. Mayr and M. Shirmohammadi and P. Totzke},
booktitle={ICALP},
year={2019}
}
We study countably infinite Markov decision processes with Buchi objectives, which ask to visit a given subset of states infinitely often. A question left open by T.P. Hill in 1979 is whether there always exist $\varepsilon$-optimal Markov strategies, i.e., strategies that base decisions only on the current state and the number of steps taken so far. We provide a negative answer to this question by constructing a non-trivial counterexample. On the other hand, we show that Markov strategies with… Expand
8 Citations

#### Figures, Tables, and Topics from this paper

Büchi Objectives in Countable MDPs
We study countably infinite Markov decision processes with Büchi objectives, which ask to visit a given subset F of states infinitely often. A question left open by T.P. Hill in 1979 [10] is whetherExpand
How to Play in Infinite MDPs
Markov decision processes (MDPs) are a standard model for dynamic systems that exhibit both stochastic and nondeterministic behavior. For MDPs with finite state space it is known that for a wideExpand
How to Play in Infinite MDPs
• 2020
Markov decision processes (MDPs) are a standard model for dynamic systems that exhibit both stochastic and nondeterministic behavior. For MDPs with finite state space it is known that for a wideExpand
Transience in Countable MDPs
• Computer Science, Mathematics
• CONCUR
• 2021
The Transience objective is not to visit any state infinitely often. While this is not possible in any finite Markov Decision Process (MDP), it can be satisfied in countably infinite ones, e.g., ifExpand
Strategy Complexity of Parity Objectives in Countable MDPs
• Computer Science, Mathematics
• CONCUR
• 2020
A complete picture of the exact strategy complexity of $\varepsilon$-optimal strategies (and optimal strategies, where they exist) for all subclasses of parity objectives in the Mostowski hierarchy is provided. Expand
Taming denumerable Markov decision processes with decisiveness
• Computer Science
• ArXiv
• 2020
This paper explores how to extend the notion of decisiveness to Markov decision processes, and whether these notions yield model checking procedures concerning the infimum and supremum probabilities of reachability properties. Expand
Symbolic controller synthesis for Büchi specifications on stochastic systems
• Computer Science, Engineering
• HSCC
• 2020
The original policy synthesis problem for continuous-state controlled Markov processes evolving in discrete time is reduced to a Büchi game under a fairness assumption and upper and lower bounds on winning sets are characterized as nested fixed point expressions in the μ-calculus. Expand
Strategy Complexity of Mean Payoff, Total Payoff and Point Payoff Objectives in Countable MDPs
• Computer Science, Mathematics
• CONCUR
• 2021
A complete picture is established of the strategy complexity of countably infinite Markov decision processes with real-valued transition rewards, i.e., how much memory is necessary and sufficient for ε-optimal (resp. optimal) strategies. Expand

#### References

SHOWING 1-10 OF 23 REFERENCES
Büchi Objectives in Countable MDPs
We study countably infinite Markov decision processes with Büchi objectives, which ask to visit a given subset F of states infinitely often. A question left open by T.P. Hill in 1979 [10] is whetherExpand
Parity objectives in countable MDPs
• Mathematics, Computer Science
• 2017 32nd Annual ACM/IEEE Symposium on Logic in Computer Science (LICS)
• 2017
We study countably infinite MDPs with parity objectives, and special cases with a bounded number of colors in the Mostowski hierarchy (including reachability, safety, Büchi and co-Büchi).
On the Existence of Good Markov Strategies
In contrast to the known fact that there are gambling problems based on a finite state space for which no stationary family of strategies is at all good, in every such problem there always existExpand
On the existence of stationary optimal strategies
The question with which this paper is concerned is roughly speaking: In a gambling situation or dynamical programming situation, are strategies that take the past into account any better than thoseExpand
A survey of stochastic ω-regular games
• Computer Science, Mathematics
• J. Comput. Syst. Sci.
• 2012
We summarize classical and recent results about two-player games played on graphs with @w-regular objectives. These games have applications in the verification and synthesis of reactive systems.Expand
Quantitative stochastic parity games
• Mathematics, Computer Science
• SODA '04
• 2004
The existence of optimal pure memoryless strategies together with the polynomial-time solution for the one-player case implies that the quantitative two-player stochastic parity game problem is in NP ∩ co-NP, which generalizes a result of Condon for Stochastic games with reachability objectives. Expand
Principles of model checking
• Computer Science
• 2008
Principles of Model Checking offers a comprehensive introduction to model checking that is not only a text suitable for classroom use but also a valuable reference for researchers and practitioners in the field. Expand
A survey of computational complexity results in systems and control
• Computer Science
• Autom.
• 2000
This paper considers problems related to stability or stabilizability of linear systems with parametric uncertainty, robust control, time-varying linear systems, nonlinear and hybrid systems, and stochastic optimal control. Expand
Determinacy and Optimal Strategies in Stochastic Games
Zabývame se determinovanosti stochastických tahových her s uplnou informaci a s výhernimi cili dosažitelnosti, bezpecnosti a Buchiho akceptacni podminky. Odděleně rozebirame situaci pro konecněExpand
goal problems in gambling theory
A short introduction to goal problems in abstract gambling theory is given, along with statementes of some of the main theorems and a number of examples, open problems and references. Emphasis is onExpand