Multi-objective Robust Strategy Synthesis for Interval Markov Decision Processes

@article{Hahn2017MultiobjectiveRS,
  title={Multi-objective Robust Strategy Synthesis for Interval Markov Decision Processes},
  author={Ernst Moritz Hahn and Vahid Hashemi and Holger Hermanns and Morteza Lahijanian and Andrea Turrini},
  journal={ArXiv},
  year={2017},
  volume={abs/1706.06875}
}
Interval Markov decision processes (IMDPs) generalise classical MDPs by having interval-valued transition probabilities. They provide a powerful modelling tool for probabilistic systems with an additional variation or uncertainty that prevents the knowledge of the exact transition probabilities. In this paper, we consider the problem of multi-objective robust strategy synthesis for interval MDPs, where the aim is to find a robust strategy that guarantees the satisfaction of multiple properties… 
View PDF on arXiv
View With Semantic Reader
29 Citations
Highly Influential Citations
2
Background Citations
15
Methods Citations
1

29 Citations

Interval Markov Decision Processes with Multiple Objectives
TLDR
This article considers Interval Markov decision processes (IMDPs), which generalise classical MDPs by having interval-valued transition probabilities, and investigates the problem of robust multi-objective synthesis for IMDPs and Pareto curve analysis of multi- objective queries on IM DPs and shows that the multi-Objective synthesis problem is PSPACE-hard.
Interval Markov Decision Processes with Multiple Objectives: from Robust Strategies to Pareto Curves
TLDR
This article considers Interval Markov decision processes ( IMDP s), which generalise classical MDP s by having interval-valued transition probabilities and investigates the problem of robust multi-objective synthesis for IMDP and Pareto curve analysis of multi- objective queries on IMDP, and shows that the multi-Objective synthesis problem is PSPACE -hard.
Multi-Objective Approaches to Markov Decision Processes with Uncertain Transition Parameters
TLDR
This paper presents and evaluates approaches to compute the pure Pareto optimal policies in the value vector space for bounded-parameter MDPs (BMDPs), a popular model for performance analysis and optimization of stochastic systems.
Multi-cost Bounded Tradeoff Analysis in MDP
TLDR
The need for more detailed visual presentations of results beyond Pareto curves is discussed and a first visualisation approach that exploits all the available information from the algorithm to support decision makers is presented.
Decision algorithms for modelling, optimal control and verification of probabilistic systems
TLDR
This dissertation focuses on decision algorithms for modelling and performance evaluation of probabilistic systems leveraging techniques from mathematical optimization and introduces a novel stochastic model, Uncertain weighted Markov Decision Processes (UwMDPs), so as to capture quantities like preferences or priorities in a nondeterministic scenario with uncertainties.
Convex Optimization for Parameter Synthesis in MDPs
TLDR
Two approaches that iteratively obtain locally optimal solutions of parametric MDPs and a sequential convex programming method are developed that improve the runtime and scalability by multiple orders of magnitude compared to black-box CCP and SCP.
Multi-Objective Controller Synthesis with Uncertain Human Preferences
TLDR
This work formalizes the notion of uncertain human preferences, and presents a novel approach that accounts for this uncertainty in the context of multi-objective controller synthesis for Markov decision processes (MDPs).
Multi-cost Bounded Reachability in MDP
TLDR
The need for output beyond Pareto curves is discussed and the available information from the algorithm is exploited to support decision makers and show the algorithm’s scalability.
Multiagent Task Allocation and Planning with Multi-Objective Requirements
TLDR
This paper considers the problem of concurrently allocating LTL task sequences to a team of agents and calculating optimal task schedulers simultaneously, satisfying cost and probability thresholds, and reduces this problem to multi-objective scheduler synthesis for a team MDP structure, whose size is linear in the number of agents.
Robust Policy Synthesis for Uncertain POMDPs via Convex Optimization
TLDR
The feasibility of the approach, which provides a transformation of the problem to a convex QCQP with finitely many constraints, is demonstrated by means of several case studies that highlight typical bottlenecks for the problem.
...
...

References

SHOWING 1-10 OF 42 REFERENCES
Markov Decision Processes with Multiple Objectives
TLDR
It is shown that every Pareto-optimal point can be achieved by a memoryless strategy; however, unlike in the single-objective case, the memoryless strategies may require randomization.
A Compromise Programming Approach to multiobjective Markov Decision Processes
TLDR
A reference point method based on the optimization of a weighted ordered weighted average (WOWA) of individual disachievements is introduced and it is shown that the resulting notion of optimal policy does not satisfy the Bellman principle and depends on the initial state.
Synthesis for PCTL in Parametric Markov Decision Processes
TLDR
This paper studies the synthesis problem for PCTL in PMDPs, and synthesises the parameter valuations under which F is true, using existing decision procedures to check whether F holds on each of the Markov processes represented by the hyper-rectangle.
Reachability analysis of uncertain systems using bounded-parameter Markov decision processes
Robust Control of Markov Decision Processes with Uncertain Transition Matrices
TLDR
This work considers a robust control problem for a finite-state, finite-action Markov decision process, where uncertainty on the transition matrices is described in terms of possibly nonconvex sets, and shows that perfect duality holds for this problem, and that it can be solved with a variant of the classical dynamic programming algorithm, the "robust dynamic programming" algorithm.
Robust control of uncertain Markov Decision Processes with temporal logic specifications
TLDR
A procedure from probabilistic model checking is used to combine the system model with an automaton representing the specification and this new MDP is transformed into an equivalent form that satisfies assumptions for stochastic shortest path dynamic programming.
Approximation of Lorenz-Optimal Solutions in Multiobjective Markov Decision Processes
TLDR
Methods to efficiently approximate the sets of Lorenz-non-dominated solutions of infinite-horizon, discounted MOMDPs are introduced, which are polynomial-sized subsets of those solutions.
Multi-Objective Model Checking of Markov Decision Processes
TLDR
It is shown that one can compute an approximate Pareto curve with respect to a set of ω-regular properties in time polynomial in the size of the MDP.
Multi-objective decision-theoretic path planning
  • A. Mouaddib
  • Computer Science, Economics
    IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004
  • 2004
TLDR
This paper transforms the problem of optimizing multi-objective decision-theoretic path planning in a Markov decision process with a multi-dimensional value function and describes techniques that allow to derive an optimal policy where it is hard to express the expected utilities, rewards and values with a numerical measure.
Bounded-parameter Markov decision processes
...
...