Relaxations of Weakly Coupled Stochastic Dynamic Programs

  title={Relaxations of Weakly Coupled Stochastic Dynamic Programs},
  author={Daniel Adelman and Adam J. Mersereau},
  journal={Oper. Res.},
We consider a broad class of stochastic dynamic programming problems that are amenable to relaxation via decomposition. These problems comprise multiple subproblems that are independent of each other except for a collection of coupling constraints on the action space. We fit an additively separable value function approximation using two techniques, namely, Lagrangian relaxation and the linear programming (LP) approach to approximate dynamic programming. We prove various results comparing the… 

Figures and Tables from this paper

Weakly Coupled Dynamic Program: Information and Lagrangian Relaxations

This work proposes a computationally tractable method based on information relaxation, and shows it provides a valid dual bound and its performance has a uniform bound regardless of the number of subproblems.

On the Strength of Relaxations of Weakly Coupled Stochastic Dynamic Programs

Many sequential decision problems have a weakly coupled structure in that a set of linking constraints couples an otherwise independent collection of subproblems. This structure arises in a wide

Discrete Nonlinear Optimization by State-Space Decompositions

A decomposition approach for binary optimization problems with nonlinear objectives and linear constraints, and it is shown that the associated transition graphs can be related by a mixed-integer linear program (MILP) so as to produce exact solutions to the original nonlinear problem.

Information Relaxations, Duality, and Convex Stochastic Dynamic Programs

This paper studies DPs that have a convex structure and considers gradient penalties that are based on first-order linear approximations of approximate value functions that can provide tight bounds for convex DPs and can be used to improve on bounds provided by other relaxations, such as Lagrangian relaxation bounds.

Integer Programing for Weakly Coupled Stochastic Dynamic Programs with Partial Information

Algorithm for problems where a decision maker has to control a system composed of several components and has access to only partial information on the state of each component, and a collection of mathematical programming formulations and algorithms which provide tractable policies and upper bounds for weakly coupled POMDPs.

Information Relaxations and Duality in Stochastic Dynamic Programs

This approach relaxes the nonanticipativity constraints that require decisions to depend only on the information available at the time a decision is made and imposes a “penalty” that punishes violations of nonant anticipativity.

Computation and Dynamic Programming

A variety of methods that are targeted towards alleviating the computational difficulties associated with dynamic programming are described, including simulation based and model free methods, the linear programming approach to approximate dynamic programming, approximate policy iteration, rollout policies, and state aggregation.

Information Relaxation Bounds for Infinite Horizon Markov Decision Processes

It is shown that any reformulation of the DP can greatly simplify solutions of the information relaxations, both in leading to finite horizon subproblems and by reducing the number of states that need to be considered in these subpro problems.

An Improved Dynamic Programming Decomposition Approach for Network Revenue Management

  • Dan Zhang
  • Computer Science
    Manuf. Serv. Oper. Manag.
  • 2011
This work considers a nonlinear nonseparable functional approximation to the value function of a dynamic programming formulation for the network revenue management (RM) problem with customer choice and shows that it leads to a tighter upper bound on optimal expected revenue than some known bounds in the literature.

Decomposable Markov Decision Processes: A Fluid Optimization Approach

This paper proposes an approximate solution approach to decomposable MDPs that is based on re-solving a fluid linear optimization formulation of the problem at each decision epoch, and proves that this fluid formulation provides a tighter bound on the optimal value function than three state-of-the-art formulations.



The Linear Programming Approach to Approximate Dynamic Programming

An efficient method based on linear programming for approximating solutions to large-scale stochastic control problems by "fits" a linear combination of pre-selected basis functions to the dynamic programming cost-to-go function.

Solving Very Large Weakly Coupled Markov Decision Processes

A technique for computing approximately optimal solutions to stochastic resource allocation problems modeled as Markov decision processes (MDPS) and describes heuristic techniques for dealing with several classes of constraints that use the solutions for individual MDPS to construct an approximate global solution.

An adaptive dynamic programming algorithm for a stochastic multiproduct batch dispatch problem

A key structural property for the decision function is proved, and this property is exploited in the development of continuous value function approximations that form the basis of an approximate dispatch rule.

The Lagrangian Relaxation Method for Solving Integer Programming Problems

This paper is a review of Lagrangian relaxation based on what has been learned in the last decade and has led to dramatically improved algorithms for a number of important problems in the areas of routing, location, scheduling, assignment and set covering.

On Constraint Sampling in the Linear Programming Approach to Approximate Dynamic Programming

A scheme that samples and imposes a subset of constraints on a linear program that has a relatively small number of variables but an intractable number of constraints is studied.

The Complexity of Optimal Queuing Network Control

It is shown that several versions of the problem of optimally controlling a simple network of queues with simple arrival and service distributions and multiple customer classes is complete for exponential time.

Generalized Lagrange Multiplier Method for Solving Problems of Optimum Allocation of Resources

The usefulness of Lagrange multipliers for optimization in the presence of constraints is not limited to differentiable functions. They can be applied to problems of maximizing an arbitrary real

A Price-Directed Approach to Stochastic Inventory/Routing

A new approach to stochastic inventory/routing that approximates the future costs of current actions using optimal dual prices of a linear program and an efficient algorithm to both generate and eliminate itineraries during solution of the linear programs and control policy are considered.

Selected Topics in Column Generation

The growing understanding of the dual point of view is emphasized, which has brought considerable progress to the column generation theory and practice, and is an ever recurring concept in "selected topics."

The LP/POMDP marriage: Optimization with imperfect information

The technique uses a master linear program to determine allocations among a set of control policies, and uses partially observable Markov decision processes (POMDPs) to determine improving policies using dual prices from the master LP.