Büchi Objectives in Countable MDPs

@inproceedings{Kiefer2019BchiOI,
  title={B{\"u}chi Objectives in Countable MDPs},
  author={S. Kiefer and R. Mayr and M. Shirmohammadi and P. Totzke},
  booktitle={ICALP},
  year={2019}
}
We study countably infinite Markov decision processes with Buchi objectives, which ask to visit a given subset of states infinitely often. A question left open by T.P. Hill in 1979 is whether there always exist $\varepsilon$-optimal Markov strategies, i.e., strategies that base decisions only on the current state and the number of steps taken so far. We provide a negative answer to this question by constructing a non-trivial counterexample. On the other hand, we show that Markov strategies with… Expand
8 Citations
Büchi Objectives in Countable MDPs
We study countably infinite Markov decision processes with Büchi objectives, which ask to visit a given subset F of states infinitely often. A question left open by T.P. Hill in 1979 [10] is whetherExpand
How to Play in Infinite MDPs
Markov decision processes (MDPs) are a standard model for dynamic systems that exhibit both stochastic and nondeterministic behavior. For MDPs with finite state space it is known that for a wideExpand
How to Play in Infinite MDPs
Markov decision processes (MDPs) are a standard model for dynamic systems that exhibit both stochastic and nondeterministic behavior. For MDPs with finite state space it is known that for a wideExpand
Transience in Countable MDPs
The Transience objective is not to visit any state infinitely often. While this is not possible in any finite Markov Decision Process (MDP), it can be satisfied in countably infinite ones, e.g., ifExpand
Strategy Complexity of Parity Objectives in Countable MDPs
TLDR
A complete picture of the exact strategy complexity of $\varepsilon$-optimal strategies (and optimal strategies, where they exist) for all subclasses of parity objectives in the Mostowski hierarchy is provided. Expand
Taming denumerable Markov decision processes with decisiveness
TLDR
This paper explores how to extend the notion of decisiveness to Markov decision processes, and whether these notions yield model checking procedures concerning the infimum and supremum probabilities of reachability properties. Expand
Symbolic controller synthesis for Büchi specifications on stochastic systems
TLDR
The original policy synthesis problem for continuous-state controlled Markov processes evolving in discrete time is reduced to a Büchi game under a fairness assumption and upper and lower bounds on winning sets are characterized as nested fixed point expressions in the μ-calculus. Expand
Strategy Complexity of Mean Payoff, Total Payoff and Point Payoff Objectives in Countable MDPs
TLDR
A complete picture is established of the strategy complexity of countably infinite Markov decision processes with real-valued transition rewards, i.e., how much memory is necessary and sufficient for ε-optimal (resp. optimal) strategies. Expand

References

SHOWING 1-10 OF 23 REFERENCES
Büchi Objectives in Countable MDPs
We study countably infinite Markov decision processes with Büchi objectives, which ask to visit a given subset F of states infinitely often. A question left open by T.P. Hill in 1979 [10] is whetherExpand
Parity objectives in countable MDPs
We study countably infinite MDPs with parity objectives, and special cases with a bounded number of colors in the Mostowski hierarchy (including reachability, safety, Büchi and co-Büchi).
On the Existence of Good Markov Strategies
In contrast to the known fact that there are gambling problems based on a finite state space for which no stationary family of strategies is at all good, in every such problem there always existExpand
On the existence of stationary optimal strategies
The question with which this paper is concerned is roughly speaking: In a gambling situation or dynamical programming situation, are strategies that take the past into account any better than thoseExpand
A survey of stochastic ω-regular games
We summarize classical and recent results about two-player games played on graphs with @w-regular objectives. These games have applications in the verification and synthesis of reactive systems.Expand
Quantitative stochastic parity games
TLDR
The existence of optimal pure memoryless strategies together with the polynomial-time solution for the one-player case implies that the quantitative two-player stochastic parity game problem is in NP ∩ co-NP, which generalizes a result of Condon for Stochastic games with reachability objectives. Expand
Principles of model checking
TLDR
Principles of Model Checking offers a comprehensive introduction to model checking that is not only a text suitable for classroom use but also a valuable reference for researchers and practitioners in the field. Expand
A survey of computational complexity results in systems and control
TLDR
This paper considers problems related to stability or stabilizability of linear systems with parametric uncertainty, robust control, time-varying linear systems, nonlinear and hybrid systems, and stochastic optimal control. Expand
Determinacy and Optimal Strategies in Stochastic Games
Zabývame se determinovanosti stochastických tahových her s uplnou informaci a s výhernimi cili dosažitelnosti, bezpecnosti a Buchiho akceptacni podminky. Odděleně rozebirame situaci pro konecněExpand
goal problems in gambling theory
A short introduction to goal problems in abstract gambling theory is given, along with statementes of some of the main theorems and a number of examples, open problems and references. Emphasis is onExpand
...
1
2
3
...