# Risk-Sensitive Reinforcement Learning

@article{Shen2014RiskSensitiveRL, title={Risk-Sensitive Reinforcement Learning}, author={Yun Shen and Michael J. Tobia and T. Sommer and K. Obermayer}, journal={Neural Computation}, year={2014}, volume={26}, pages={1298-1328} }

We derive a family of risk-sensitive reinforcement learning methods for agents, who face sequential decision-making tasks in uncertain environments. By applying a utility function to the temporal difference (TD) error, nonlinear transformations are effectively applied not only to the received rewards but also to the true transition probabilities of the underlying Markov decision process. When appropriate utility functions are chosen, the agents’ behaviors express key features of human behavior… Expand

#### Topics from this paper

#### 134 Citations

Risk-Sensitive Inverse Reinforcement Learning via Gradient Methods

- Computer Science, Mathematics
- ArXiv
- 2017

This work addresses the problem of inverse reinforcement learning in Markov decision processes where the agent is risk-sensitive by making use of models of human decisionmaking having their origins in behavioral psychology, behavioral economics, and neuroscience. Expand

Risk-sensitive Inverse Reinforcement Learning via Coherent Risk Models

- Computer Science
- Robotics: Science and Systems
- 2017

Comparisons of the RiskSensitive (RS) IRL approach with a risk-neutral model show that the RS-IRL framework more accurately captures observed participant behavior both qualitatively and quantitatively. Expand

Risk-sensitive inverse reinforcement learning via semi- and non-parametric methods

- Computer Science
- Int. J. Robotics Res.
- 2018

Comparisons with a risk-neutral model show that the RS-IRL framework more accurately captures observed participant behavior both qualitatively and quantitatively, especially in scenarios where catastrophic outcomes such as collisions can occur. Expand

Gradient-based inverse risk-sensitive reinforcement learning

- Computer Science
- 2017 IEEE 56th Annual Conference on Decision and Control (CDC)
- 2017

This work proposes a gradient-based inverse reinforcement learning algorithm that minimizes a loss function defined on the observed behavior that is compatible with Markov decision processes where the agent is risksensitive. Expand

Inverse Risk-Sensitive Reinforcement Learning

- Computer Science, Mathematics
- IEEE Transactions on Automatic Control
- 2020

A risk-sensitive reinforcement learning algorithm with convergence guarantees that makes use of coherent risk metrics and models of human decision-making which have their origins in behavioral psychology and economics is presented. Expand

Epistemic Risk-Sensitive Reinforcement Learning

- Computer Science, Mathematics
- ESANN
- 2020

A framework for interacting with uncertain environments in reinforcement learning by leveraging preferences in the form of utility functions that can be risk-averse, risk-neutral or risk-taking depending on the parameter choice is developed. Expand

Risk-Aware Transfer in Reinforcement Learning using Successor Features

- Computer Science
- ArXiv
- 2021

This paper addresses the problem of risk-aware policy transfer between tasks in a common domain that differ only in their reward functions, in which risk is measured by the variance of reward streams, and develops risk-awareness features that integrate seamlessly within the RL framework and inherit the superior task generalization ability of SFs. Expand

Risk-Sensitive Reinforcement Learning: A Constrained Optimization Viewpoint

- Computer Science, Mathematics
- ArXiv
- 2018

This article focuses on the combination of risk criteria and reinforcement learning in a constrained optimization framework, i.e., a setting where the goal to find a policy that optimizes the usual objective of infinite-horizon discounted/average cost, while ensuring that an explicit risk constraint is satisfied. Expand

A Scheme for Dynamic Risk-Sensitive Sequential Decision Making

- Computer Science
- ArXiv
- 2019

It is shown that most risk measures can be estimated using return variance and, by virtue of the state-augmentation transformation, practical problems modeled by Markov decision processes with stochastic rewards can be solved in a risk-sensitive scenario. Expand

RAIL: Risk-Averse Imitation Learning

- Computer Science, Mathematics
- AAMAS
- 2018

The proposed RAIL algorithm appears as a potent alternative to GAIL for improved reliability in risk-sensitive applications by minimizing tail risk within the GAIL framework by quantifying tail risk by the Conditional-Value-at-Risk of trajectories and developing the Risk-Averse Imitation Learning (RAIL) algorithm. Expand

#### References

SHOWING 1-10 OF 39 REFERENCES

Risk-Sensitive Reinforcement Learning

- Mathematics, Computer Science
- Machine Learning
- 2004

This risk-sensitive reinforcement learning algorithm is based on a very different philosophy and reflects important properties of the classical exponential utility framework, but avoids its serious drawbacks for learning. Expand

Consideration of Risk in Reinforcement Learning

- Mathematics, Computer Science
- ICML
- 1994

The purpose of this paper is to draw the reader's attention to the problems of the expected value criterion in Markov decision processes and to give Dynamic Programming algorithms for an alternative criterion, namely the minimax criterion. Expand

Neural Prediction Errors Reveal a Risk-Sensitive Reinforcement-Learning Process in the Human Brain

- Psychology, Medicine
- The Journal of Neuroscience
- 2012

Humans and animals are exquisitely, though idiosyncratically, sensitive to risk or variance in the outcomes of their actions. Economic, psychological, and neural aspects of this are well studied when… Expand

Risk-Sensitive Optimal Feedback Control Accounts for Sensorimotor Behavior under Uncertainty

- Computer Science, Medicine
- PLoS Comput. Biol.
- 2010

It is shown that subjects change their movement strategy pessimistically in the face of increased uncertainty in accord with the predictions of a risk-averse optimal controller, suggesting that risk-sensitivity is a fundamental attribute that needs to be incorporated into optimal feedback control models. Expand

Human Insula Activation Reflects Risk Prediction Errors As Well As Risk

- Psychology, Medicine
- The Journal of Neuroscience
- 2008

Using functional imaging during a simple gambling task, it is shown that an early-onset activation in the human insula correlates significantly with risk prediction error and that its time course is consistent with a role in rapid updating. Expand

Q-Learning for Risk-Sensitive Control

- Computer Science
- Math. Oper. Res.
- 2002

We propose for risk-sensitive control of finite Markov chains a counterpart of the popular Q-learning algorithm for classical Markov decision processes. The algorithm is shown to converge with… Expand

Risk-Sensitivity in Sensorimotor Control

- Psychology, Medicine
- Front. Hum. Neurosci.
- 2011

Evidence that humans exhibit risk-sensitivity in their motor behaviors, thereby demonstrating sensitivity to the variability of “motor costs,” is reviewed and it is concluded that risk-Sensitivity is an important concept in understanding individual motor behavior under uncertainty. Expand

Advances in prospect theory: Cumulative representation of uncertainty

- Economics
- 1992

We develop a new version of prospect theory that employs cumulative rather than separable decision weights and extends the theory in several respects. This version, called cumulative prospect theory,… Expand

The economics of risk and time

- Economics
- 2001

This book updates and advances the theory of expected utility as applied to risk analysis and financial decision making. Von Neumann and Morgenstern pioneered the use of expected utility theory in… Expand

Risk-averse dynamic programming for Markov decision processes

- Mathematics, Computer Science
- Math. Program.
- 2010

The concept of a Markov risk measure is introduced and it is used to formulate risk-averse control problems for two Markov decision models: a finite horizon model and a discounted infinite horizon model. Expand