Optimal Control of Conditional Value-at-Risk in Continuous Time

@article{Miller2017OptimalCO,
  title={Optimal Control of Conditional Value-at-Risk in Continuous Time},
  author={Christopher W. Miller and Insoon Yang},
  journal={SIAM J. Control. Optim.},
  year={2017},
  volume={55},
  pages={856-884}
}
We consider continuous-time stochastic optimal control problems featuring Conditional Value-at-Risk (CVaR) in the objective. The major difficulty in these problems arises from time-inconsistency, which prevents us from directly using dynamic programming. To resolve this challenge, we convert to an equivalent bilevel optimization problem in which the inner optimization problem is standard stochastic control. Furthermore, we provide conditions under which the outer objective function is convex… 

Figures from this paper

Nonlinear PDE Approach to Time-Inconsistent Optimal Stopping

TLDR
A novel method is presented for solving a class of time-inconsistent optimal stopping problems by reducing them to a family of standard stochastic optimal control problems by converting an optimal stopping problem with a non-linear function of the expected stopping time into optimization over an auxiliary value function for a standard stoChastic control problem with an additional state variable.

Stochastic Control of Optimized Certainty Equivalents

TLDR
This work considers stochastic optimal control problems where the objective criterion is given by an OCE risk measure, and shows that the value of the risk minimization problem can be characterized via the viscosity solution of a Hamilton--Jacobi--Bellman--Issacs equation.

A Stochastic Primal-Dual Method for Optimization with Conditional Value at Risk Constraints

TLDR
A first-order primal-dual subgradient method to optimize risk-constrained risk-penalized optimization problems, where risk is modeled via the popular conditional value at risk measure, surprisingly obviates the need for a priori bounds or complex adaptive bounding schemes for dual variables assumed in many prior works.

Conditional Value at Risk Sensitive Optimization via Subgradient Methods

We study a first-order primal-dual subgradient method to optimize risk-constrained risk-penalized optimization problems, where risk is modeled via the popular conditional value at risk (CVaR)

Risk-sensitive safety analysis using Conditional Value-at-Risk

TLDR
This paper develops a safety analysis method for stochastic systems that is sensitive to the possibility and severity of rare harmful outcomes and provides a novel, theoretically guaranteed, parameter-dependent upper bound to the CVaR of a maximum cost without the need to augment the state space.

Safety-Aware Optimal Control of Stochastic Systems Using Conditional Value-at-Risk

TLDR
A novel measure of safety risk is introduced using the conditional value-at-risk and a set distance to formulate a safety risk-constrained optimal control problem, and an extremal representation of the safety risk measure provides a computationally tractable dynamic programming solution.

Variance-Constrained Risk Sharing in Stochastic Systems

TLDR
This paper proposes a new continuous-time first best contract framework that has the ability to explicitly limit the agent's risks by formulating the contract design problem as a mean-variance constrained stochastic optimal control problem and develops a dynamic programming-based solution approach.

On the dynamic representation of some time-inconsistent risk measures in a Brownian filtration

TLDR
This work shows that in a Brownian filtration the “Optimized Certainty Equivalent” risk measures of Ben-Tal and Teboulle can be computed through PDE techniques, i.e. dynamically, and obtains a dynamic programming principle and stochastic control techniques, along with the theory of viscosity solutions, which must adapt to cover the present singular situation.

Risk-averse Distributional Reinforcement Learning: A CVaR Optimization Approach

TLDR
This paper improves the CVaR Value Iteration algorithm in a way that reduces computational complexity of the original algorithm from polynomial to linear time, and proposes an approximate Q-learning algorithm by reformulating theCVaR Temporal Difference update rule as a loss function which is later used in a deep learning context.

Risk-sensitive safety specifications for stochastic systems using Conditional Value-at-Risk

TLDR
A safety analysis method that facilitates a tunable balance between worst-case and risk-neutral perspectives and enables tractable policy synthesis for a class of linear systems is proposed.

References

SHOWING 1-10 OF 64 REFERENCES

Nonlinear PDE Approach to Time-Inconsistent Optimal Stopping

TLDR
A novel method is presented for solving a class of time-inconsistent optimal stopping problems by reducing them to a family of standard stochastic optimal control problems by converting an optimal stopping problem with a non-linear function of the expected stopping time into optimization over an auxiliary value function for a standard stoChastic control problem with an additional state variable.

Dynamic Approaches for Some Time Inconsistent Problems

TLDR
The main contribution of this work is the introduction of the idea of the "dynamic utility" under which the original time inconsistent problem (under the fixed utility) becomes a time consistent one.

Risk-Averse Control of Diffusion Processes

We introduce the concept of a continuous-time Markov risk measure for controlled diffusion processes. We use it to formulate a risk-averse control problem where the costs are accumulated during the

Minimum Average Value-at-Risk for Finite Horizon Semi-Markov Decision Processes in Continuous Time

TLDR
It is proved that the value function is the unique solution in a metric space to the optimality equation when one more condition is imposed, which plays a key role for the algorithm complexity analysis and the policy improvemen...

Variance-Constrained Risk Sharing in Stochastic Systems

TLDR
This paper proposes a new continuous-time first best contract framework that has the ability to explicitly limit the agent's risks by formulating the contract design problem as a mean-variance constrained stochastic optimal control problem and develops a dynamic programming-based solution approach.

A Convex Analytic Approach to Risk-Aware Markov Decision Processes

TLDR
In classical Markov decision process (MDP) theory, a policy is searched for that minimizes the expected infinite horizon discounted cost in two cases, the expected utility framework, and conditional value-at-risk, a popular coherent risk measure.

A General Theory of Markovian Time Inconsistent Stochastic Control Problems

We develop a theory for stochastic control problems which, in various ways, are time inconsistent in the sense that they do not admit a Bellman optimality principle. We attach these problems by

Convex Control Systems and Convex Optimal Control Problems With Constraints

TLDR
A conceptual computational approach based on gradient-type methods and proximal point techniques is proposed, showing that, for suitable cost functionals and constraints, optimal control problems for these classes of systems correspond to convex optimization problems.

Optimal Stochastic Control, Stochastic Target Problems, and Backward SDE

Preface.- 1. Conditional Expectation and Linear Parabolic PDEs.- 2. Stochastic Control and Dynamic Programming.- 3. Optimal Stopping and Dynamic Programming.- 4. Solving Control Problems by

Risk-Sensitive Control and Dynamic Games for Partially Observed Discrete-Time Nonlinear Systems

In this paper we solve a finite-horizon partially observed risk-sensitive stochastic optimal control problem for discrete-time nonlinear systems and obtain small noise and small risk limits. The
...