Structured Solution Methods for Non-Markovian Decision Processes


Markov Decision Processes (MDPs), currently a popular method for modeling and solving decision theoretic planning problems, are limited by the Markovian assumption: rewards and dynamics depend on the current state only, and not on previous history. Non-Markovian decision processes (NMDPs) can also be defined, but then the more tractable solution techniques developed for MDP’s cannot be directly applied. In this paper, we show how an NMDP, in which temporal logic is used to specify history dependence, can be automatically converted into an equivalent MDP by adding appropriate temporal variables. The resulting MDP can be represented in a structured fashion and solved using structured policy construction methods. In many cases, this offers significant computational advantagesover previous proposals for solving NMDPs.

Extracted Key Phrases

Cite this paper

@inproceedings{Bacchus1997StructuredSM, title={Structured Solution Methods for Non-Markovian Decision Processes}, author={Fahiem Bacchus and Craig Boutilier and Adam J. Grove}, booktitle={AAAI/IAAI}, year={1997} }