# Survey of linear programming for standard and nonstandard Markovian control problems. Part II: Applications

@article{Kallenberg1994SurveyOL, title={Survey of linear programming for standard and nonstandard Markovian control problems. Part II: Applications}, author={Lodewijk C. M. Kallenberg}, journal={Zeitschrift f{\"u}r Operations Research}, year={1994}, volume={40}, pages={127-143} }

This paper deals with some applications of Markov decision models for which the linear programming method is efficient. These models are replacement models (with the optimal stopping problem as special case), separable models (including the inventory model as special case) and the multi-armed bandit model. In the companion paper “Survey of linear programming for standard and nonstandard Markovian control problems. Part I: Theory”, general linear programming methods are discussed. These linear…

## 9 Citations

### Linear Programming Approximations for Markov Control Processes in Metric Spaces

- Mathematics, Computer ScienceProceedings of the 36th IEEE Conference on Decision and Control
- 1997

A general framework to analyze the convergence of linear-programming approximations for Markov control processes in metric spaces is developed, under which the control problem’s optimal value can be approximated by a sequence of finite-dimensional linear programs.

### Polynomial approximation method for stochastic programming.

- Computer Science
- 2009

This thesis applies SAA, polynomial approximation method and then steepest descent method in combination to solve the large-scale problems effectively and efficiently.

### Finding the K best policies in a finite-horizon Markov decision process

- Computer Science, MathematicsEur. J. Oper. Res.
- 2006

### Derman’s book as inspiration: some results on LP for MDPs

- MathematicsAnnals of Operations Research
- 2012

In 1976 I was looking for a suitable subject for my PhD thesis. My thesis advisor Arie Hordijk and I found a lot of inspiration in Derman’s book (Finite state Markovian decision processes, Academic…

### Derman’s book as inspiration: some results on LP for MDPs

- MathematicsAnn. Oper. Res.
- 2013

This article will describe some results in this area on the following topics: (1) MDPs with the average reward criterion; (2) additional constraints; (3) applications.

### Optimal Policies for Multi-server Non-preemptive Priority Queues

- BusinessQueueing Syst. Theory Appl.
- 2002

A linear programming approach is used to find and evaluate the performance of an asymptotically optimal policy in the setting of exponential service and inter-arrival times in a multi-server non-preemptive queue.

### Risk-Aware Scheduling of Dual Criticality Job Systems Using Demand Distributions

- Computer ScienceLeibniz Trans. Embed. Syst.
- 2018

A probabilistic framework for MC scheduling is developed, where feasibility is defined in terms of (chance) constraints on the probabilities that Lo and Hi jobs meet their deadlines, and those paths are dependent upon the set of execution scenarios and the given demand distributions.

### High assurance real-time systems : scheduling, budgeting, and safety

- Computer Science
- 2019

This dissertation incorporates software failure rates into the scheduling problem, and synthesizes feasible scheduling policies under which the input task set will satisfy certain failure rate requirements, and model the task scheduling problem as a path-Constrained Markov Decision Process (CMDP).

### Risk-Averse Control of Undiscounted Transient Markov Models

- MathematicsSIAM J. Control. Optim.
- 2014

This work uses Markov risk measures to formulate a risk-averse version of the undiscounted total cost problem for a transient controlled Markov process and derives risk- averse dynamic programming equations satisfied by the optimal policy.

## References

SHOWING 1-10 OF 18 REFERENCES

### Survey of linear programming for standard and nonstandard Markovian control problems. Part I: Theory

- MathematicsMath. Methods Oper. Res.
- 1994

This paper gives an overview of linear programming methods for solving standard and nonstandard Markovian control problems, and a particular class of stochastic games.

### Linear Programming Solutions for Separable Markovian Decision Problems

- Mathematics
- 1967

This paper is concerned with the linear programming solutions to sequential decision (or control) problems in which the stochastic element is Markovian and in which the objective is to minimize the…

### Linear programming and finite Markovian control problems

- Mathematics
- 1983

This text is a revised version of the author's thesis for the University of Leiden and is mainly concerned with the theory of finite Markov decision problems. Such problems are those where a decision…

### Myopic Solutions of Markov Decision Processes and Stochastic Games

- EconomicsOper. Res.
- 1981

Sufficient conditions are presented for a Markov decision process to have a myopic optimum and for a stochastic game to possess a myopic equilibrium point. An optimum or an equilibrium point is said…

### Separable Markovian decision problems

- Mathematics
- 1992

SummarySeparable Markovian decision problems have the property that for certain pairs (i, a) of a statei and an actiona: (i) the immediate reward is the sum of terms due to the current state and…

### Extensions of the multiarmed bandit problem: The discounted case

- Business
- 1985

A reformulation of the bandit problem yields the tax problem, which includes Klimov's waiting time problem, and an index rule is derived for the case where new machines arrive randomly.

### The Multi-Armed Bandit Problem: Decomposition and Computation

- MathematicsMath. Oper. Res.
- 1987

It is shown that an approximate largest-index rule yields an approximately optimal policy for the N-project problem, and more efficient methods of computing the indices on-line and/or for sparse transition matrices in large state spaces than have been suggested heretofore.

### Bandit processes and dynamic allocation indices

- Mathematics
- 1979

The paper aims to give a unified account of the central concepts in recent work on bandit processes and dynamic allocation indices; to show how these reduce some previously intractable problems to…

### Stochastic games with state independent transitions and separable rewards

- Computer Science
- 1984

For the class of stochastic games with separable reward functions and state independent transitions it is proved that the ordered field property holds. A solution of these stochastic games can easily…

### A Note on M. N. Katehakis' and Y.-R. Chen's Computation of the Gittins Index

- MathematicsMath. Oper. Res.
- 1986

It is shown that instead of the Kv linear programs for project v also one parametric linear program with the same dimensions can be solved.