Anytime Guarantees for Reachability in Uncountable Markov Decision Processes

  title={Anytime Guarantees for Reachability in Uncountable Markov Decision Processes},
  author={Kush Grover and Jan Křet{\'i}nsk{\'y} and Tobias Meggendorfer and Maximilian Weininger},
  booktitle={International Conference on Concurrency Theory},
We consider the problem of approximating the reachability probabilities in Markov decision processes (MDP) with uncountable (continuous) state and action spaces. While there are algorithms that, for special classes of such MDP, provide a sequence of approximations converging to the true value in the limit, our aim is to obtain an algorithm with guarantees on the precision of the approximation. As this problem is undecidable in general, assumptions on the MDP are necessary. Our main contribution… 

Figures from this paper



Verification of Markov Decision Processes Using Learning Algorithms

A general framework for applying machine-learning algorithms to the verification of Markov decision processes (MDPs) and focuses on probabilistic reachability, which is a core property for verification, and is illustrated through two distinct instantiations.

Interval iteration algorithm for MDPs and IMDPs

Quantitative automata-based controller synthesis for non-autonomous stochastic hybrid systems

The contribution shows that Markov processes that are defined over an uncountable state space and embedding non-determinism in the shape of a control structure can be sufficiently tackled with history-independent Markov policies.

Bounded real-time dynamic programming: RTDP with monotone upper bounds and performance guarantees

A new algorithm, Bounded RTDP, is introduced, which can produce partial policies with strong performance guarantees while only touching a fraction of the state space, even on problems where other algorithms would have to visit the full state space.

Optimistic Value Iteration

This paper obtains a lower bound via standard value iteration, uses the result to “guess” an upper bound, and proves the latter’s correctness, and presents this optimistic value iteration approach for computing reachability probabilities as well as expected rewards.

ProbReach: verified probabilistic delta-reachability for stochastic hybrid systems

The capabilities of ProbReach are introduced, a probabilistic version of delta-reachability that is suited for hybrid systems whose stochastic behaviour is given in terms of random initial conditions is implemented, and results for several benchmarks involving highly non-linear hybrid systems are presented.

Bounded Verification of Reachability of Probabilistic Hybrid Systems

This paper focuses on polyhedral dynamical systems to model continuous dynamics and computation of the exact minimum/maximum probability of reachability within k discrete steps in a polyhedral probabilistic hybrid system.

Ensuring the Reliability of Your Model Checker: Interval Iteration for Markov Decision Processes

This paper presents interval iteration techniques for computing expected accumulated weights (or costs), a considerably broader class of properties, and proposes topological interval iteration, which increases efficiency using a model decomposition into strongly connected components.