# Multi-objective Robust Strategy Synthesis for Interval Markov Decision Processes

@article{Hahn2017MultiobjectiveRS, title={Multi-objective Robust Strategy Synthesis for Interval Markov Decision Processes}, author={Ernst Moritz Hahn and Vahid Hashemi and Holger Hermanns and Morteza Lahijanian and Andrea Turrini}, journal={ArXiv}, year={2017}, volume={abs/1706.06875} }

Interval Markov decision processes (IMDPs) generalise classical MDPs by having interval-valued transition probabilities. They provide a powerful modelling tool for probabilistic systems with an additional variation or uncertainty that prevents the knowledge of the exact transition probabilities. In this paper, we consider the problem of multi-objective robust strategy synthesis for interval MDPs, where the aim is to find a robust strategy that guarantees the satisfaction of multiple properties…

## 29 Citations

Interval Markov Decision Processes with Multiple Objectives

- Computer Science, MathematicsACM Trans. Model. Comput. Simul.
- 2019

This article considers Interval Markov decision processes (IMDPs), which generalise classical MDPs by having interval-valued transition probabilities, and investigates the problem of robust multi-objective synthesis for IMDPs and Pareto curve analysis of multi- objective queries on IM DPs and shows that the multi-Objective synthesis problem is PSPACE-hard.

Interval Markov Decision Processes with Multiple Objectives: from Robust Strategies to Pareto Curves

- Computer Science
- 2019

This article considers Interval Markov decision processes ( IMDP s), which generalise classical MDP s by having interval-valued transition probabilities and investigates the problem of robust multi-objective synthesis for IMDP and Pareto curve analysis of multi- objective queries on IMDP, and shows that the multi-Objective synthesis problem is PSPACE -hard.

Multi-Objective Approaches to Markov Decision Processes with Uncertain Transition Parameters

- Computer ScienceVALUETOOLS
- 2017

This paper presents and evaluates approaches to compute the pure Pareto optimal policies in the value vector space for bounded-parameter MDPs (BMDPs), a popular model for performance analysis and optimization of stochastic systems.

Multi-cost Bounded Tradeoff Analysis in MDP

- Computer ScienceJournal of Automated Reasoning
- 2020

The need for more detailed visual presentations of results beyond Pareto curves is discussed and a first visualisation approach that exploits all the available information from the algorithm to support decision makers is presented.

Decision algorithms for modelling, optimal control and verification of probabilistic systems

- Computer Science
- 2017

This dissertation focuses on decision algorithms for modelling and performance evaluation of probabilistic systems leveraging techniques from mathematical optimization and introduces a novel stochastic model, Uncertain weighted Markov Decision Processes (UwMDPs), so as to capture quantities like preferences or priorities in a nondeterministic scenario with uncertainties.

Convex Optimization for Parameter Synthesis in MDPs

- Computer ScienceIEEE Transactions on Automatic Control
- 2021

Two approaches that iteratively obtain locally optimal solutions of parametric MDPs and a sequential convex programming method are developed that improve the runtime and scalability by multiple orders of magnitude compared to black-box CCP and SCP.

Multi-Objective Controller Synthesis with Uncertain Human Preferences

- Computer Science2022 ACM/IEEE 13th International Conference on Cyber-Physical Systems (ICCPS)
- 2022

This work formalizes the notion of uncertain human preferences, and presents a novel approach that accounts for this uncertainty in the context of multi-objective controller synthesis for Markov decision processes (MDPs).

Multi-cost Bounded Reachability in MDP

- Computer ScienceTACAS
- 2018

The need for output beyond Pareto curves is discussed and the available information from the algorithm is exploited to support decision makers and show the algorithm’s scalability.

Multiagent Task Allocation and Planning with Multi-Objective Requirements

- Computer ScienceAAMAS
- 2021

This paper considers the problem of concurrently allocating LTL task sequences to a team of agents and calculating optimal task schedulers simultaneously, satisfying cost and probability thresholds, and reduces this problem to multi-objective scheduler synthesis for a team MDP structure, whose size is linear in the number of agents.

Robust Policy Synthesis for Uncertain POMDPs via Convex Optimization

- Computer ScienceIJCAI
- 2020

The feasibility of the approach, which provides a transformation of the problem to a convex QCQP with finitely many constraints, is demonstrated by means of several case studies that highlight typical bottlenecks for the problem.

## References

SHOWING 1-10 OF 42 REFERENCES

Markov Decision Processes with Multiple Objectives

- Computer ScienceSTACS
- 2006

It is shown that every Pareto-optimal point can be achieved by a memoryless strategy; however, unlike in the single-objective case, the memoryless strategies may require randomization.

A Compromise Programming Approach to multiobjective Markov Decision Processes

- Computer ScienceInt. J. Inf. Technol. Decis. Mak.
- 2013

A reference point method based on the optimization of a weighted ordered weighted average (WOWA) of individual disachievements is introduced and it is shown that the resulting notion of optimal policy does not satisfy the Bellman principle and depends on the initial state.

Synthesis for PCTL in Parametric Markov Decision Processes

- Mathematics, Computer ScienceNASA Formal Methods
- 2011

This paper studies the synthesis problem for PCTL in PMDPs, and synthesises the parameter valuations under which F is true, using existing decision procedures to check whether F holds on each of the Markov processes represented by the hyper-rectangle.

Reachability analysis of uncertain systems using bounded-parameter Markov decision processes

- MathematicsArtif. Intell.
- 2008

Robust Control of Markov Decision Processes with Uncertain Transition Matrices

- Computer Science, MathematicsOper. Res.
- 2005

This work considers a robust control problem for a finite-state, finite-action Markov decision process, where uncertainty on the transition matrices is described in terms of possibly nonconvex sets, and shows that perfect duality holds for this problem, and that it can be solved with a variant of the classical dynamic programming algorithm, the "robust dynamic programming" algorithm.

Robust control of uncertain Markov Decision Processes with temporal logic specifications

- Computer Science, Mathematics2012 IEEE 51st IEEE Conference on Decision and Control (CDC)
- 2012

A procedure from probabilistic model checking is used to combine the system model with an automaton representing the specification and this new MDP is transformed into an equivalent form that satisfies assumptions for stochastic shortest path dynamic programming.

Approximation of Lorenz-Optimal Solutions in Multiobjective Markov Decision Processes

- Economics, Computer ScienceAAAI
- 2013

Methods to efficiently approximate the sets of Lorenz-non-dominated solutions of infinite-horizon, discounted MOMDPs are introduced, which are polynomial-sized subsets of those solutions.

Multi-Objective Model Checking of Markov Decision Processes

- Computer Science, MathematicsLog. Methods Comput. Sci.
- 2007

It is shown that one can compute an approximate Pareto curve with respect to a set of ω-regular properties in time polynomial in the size of the MDP.

Multi-objective decision-theoretic path planning

- Computer Science, EconomicsIEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004
- 2004

This paper transforms the problem of optimizing multi-objective decision-theoretic path planning in a Markov decision process with a multi-dimensional value function and describes techniques that allow to derive an optimal policy where it is hard to express the expected utilities, rewards and values with a numerical measure.