Percentile queries in multi-dimensional Markov decision processes

@article{Randour2015PercentileQI,
  title={Percentile queries in multi-dimensional Markov decision processes},
  author={Mickael Randour and Jean-François Raskin and Ocan Sankur},
  journal={Formal Methods in System Design},
  year={2015},
  volume={50},
  pages={207-248}
}
Markov decision processes (MDPs) with multi-dimensional weights are useful to analyze systems with multiple objectives that may be conflicting and require the analysis of trade-offs. We study the complexity of percentile queries in such MDPs and give algorithms to synthesize strategies that enforce such constraints. Given a multi-dimensional weighted MDP and a quantitative payoff function f, thresholds $$v_i$$vi (one per dimension), and probability thresholds $$\alpha _i$$αi, we show how to… 

L O ] 7 D ec 2 01 6 Percentile Queries in Multi-Dimensional Markov Decision Processes ⋆

TLDR
Given a multi-dimensional weighted MDP and a quantitative payoff function f, thresholds vi, and probability thresholds αi, it is shown how to compute a single strategy to enforce that for all dimensions i, the probability of outcomes satisfying fi(ρ) ≥ vi is at least αi.

Unifying Two Views on Multiple Mean-Payoff Objectives in Markov Decision Processes

TLDR
This work considers Markov decision processes with multiple limit-average objectives with multiple mean-payoff objectives, and presents a complete characterization of the strategy complexity (in terms of memory bounds and randomization) required to solve the problem.

Simple Strategies in Multi-Objective MDPs (Technical Report)

TLDR
It is shown that checking whether a point is achievable by a pure stationary strategy is NP-complete, even for two objectives, and the authors provide an MILP encoding to solve the corresponding problem.

The Odds of Staying on Budget

TLDR
This work studies the computational complexity of deciding whether the probability of paths whose accumulated cost satisfies a Boolean combination of inequalities exceeds a given threshold, and shows that this problem is PP-complete, whereas it is hard for the PosSLP problem and in PSpace for general Markov chains.

Simple Strategies in Multi-Objective MDPs

TLDR
It is shown that checking whether a point is achievable by a pure stationary strategy is NP-complete, even for two objectives, and the author provides an MILP encoding to solve the corresponding problem.

Multi-cost Bounded Tradeoff Analysis in MDP

TLDR
The need for more detailed visual presentations of results beyond Pareto curves is discussed and a first visualisation approach that exploits all the available information from the algorithm to support decision makers is presented.

Interval Markov Decision Processes with Multiple Objectives

TLDR
This article considers Interval Markov decision processes (IMDPs), which generalise classical MDPs by having interval-valued transition probabilities, and investigates the problem of robust multi-objective synthesis for IMDPs and Pareto curve analysis of multi- objective queries on IM DPs and shows that the multi-Objective synthesis problem is PSPACE-hard.

Multidimensional beyond Worst-Case and Almost-Sure Problems for Mean-Payoff Objectives

TLDR
The multidimensional BAS threshold problem is solvable in P. This solves the infinite-memory threshold problem left open by Bruyère et al., and this complexity cannot be improved without improving the currently known complexity of classical mean-payoff games.

Interval Markov Decision Processes with Multiple Objectives: from Robust Strategies to Pareto Curves

TLDR
This article considers Interval Markov decision processes ( IMDP s), which generalise classical MDP s by having interval-valued transition probabilities and investigates the problem of robust multi-objective synthesis for IMDP and Pareto curve analysis of multi- objective queries on IMDP, and shows that the multi-Objective synthesis problem is PSPACE -hard.

Computing quantiles in Markov chains with multi-dimensional costs

TLDR
This paper presents an algorithm that allows to approximate the probabilities with arbitrary precision of the total cost, and enables it to show that a decision version of the cost problem lies in the counting hierarchy, a counting analogue to the polynomial-time hierarchy that contains the latter and is included in PSPACE.
...

References

SHOWING 1-10 OF 52 REFERENCES

Unifying Two Views on Multiple Mean-Payoff Objectives in Markov Decision Processes

TLDR
This work considers Markov decision processes with multiple limit-average objectives with multiple mean-payoff objectives, and presents a complete characterization of the strategy complexity (in terms of memory bounds and randomization) required to solve the problem.

Markov Decision Processes with Multiple Long-Run Average Objectives

TLDR
Algorithms for design exploration in MDP models with multiple long-run average objectives, including the problem if a given value vector is realizable by any strategy, are studied, and it is shown that it can be decided in polynomial time for irreducible M DPs and in NP for all MDPs.

Markov Decision Processes with Multiple Objectives

TLDR
It is shown that every Pareto-optimal point can be achieved by a memoryless strategy; however, unlike in the single-objective case, the memoryless strategies may require randomization.

Multi-Objective Model Checking of Markov Decision Processes

TLDR
It is shown that one can compute an approximate Pareto curve with respect to a set of ω-regular properties in time polynomial in the size of the MDP.

Percentile performance criteria for limiting average Markov decision processes

TLDR
The authors present a complete (and discrete) classification of both the maximal achievable target levels and of their corresponding percentiles and provide an algorithm for computing a deterministic policy corresponding to any feasible target-percentile pair.

The Odds of Staying on Budget

TLDR
This work studies the computational complexity of deciding whether the probability of paths whose accumulated cost satisfies a Boolean combination of inequalities exceeds a given threshold, and shows that this problem is PP-complete, whereas it is hard for the PosSLP problem and in PSpace for general Markov chains.

Computing Quantiles in Markov Reward Models

TLDR
The complexity of quantile queries for until properties in discrete-time finite-state Markov decision processes with nonnegative rewards on states is studied and it is shown that qualitative quantiles queries can be evaluated in polynomial time and an exponential algorithm for the evaluation of quantitative quantile query is presented.

Minimizing risk models in Markov decision processes with policies depending on target values

Abstract This paper studies the minimizing risk problems in Markov decision processes with countable state space and reward set. The objective is to find a policy which minimizes the probability

Probabilistic Goal Markov Decision Processes

TLDR
This work considers the policy that maximizes the probability of achieving a pre-determined target performance, a criterion the authors term probabilistic goal Markov decision processes, and shows that this problem is NP-hard, but can be solved using a pseudo-polynomial algorithm.
...