Risk-Sensitive Reinforcement Learning

@article{Shen2014RiskSensitiveRL,
  title={Risk-Sensitive Reinforcement Learning},
  author={Yun Shen and Michael J. Tobia and T. Sommer and K. Obermayer},
  journal={Neural Computation},
  year={2014},
  volume={26},
  pages={1298-1328}
}
We derive a family of risk-sensitive reinforcement learning methods for agents, who face sequential decision-making tasks in uncertain environments. By applying a utility function to the temporal difference (TD) error, nonlinear transformations are effectively applied not only to the received rewards but also to the true transition probabilities of the underlying Markov decision process. When appropriate utility functions are chosen, the agents’ behaviors express key features of human behavior… Expand
Risk-Sensitive Inverse Reinforcement Learning via Gradient Methods
TLDR
This work addresses the problem of inverse reinforcement learning in Markov decision processes where the agent is risk-sensitive by making use of models of human decisionmaking having their origins in behavioral psychology, behavioral economics, and neuroscience. Expand
Risk-sensitive Inverse Reinforcement Learning via Coherent Risk Models
TLDR
Comparisons of the RiskSensitive (RS) IRL approach with a risk-neutral model show that the RS-IRL framework more accurately captures observed participant behavior both qualitatively and quantitatively. Expand
Risk-sensitive inverse reinforcement learning via semi- and non-parametric methods
TLDR
Comparisons with a risk-neutral model show that the RS-IRL framework more accurately captures observed participant behavior both qualitatively and quantitatively, especially in scenarios where catastrophic outcomes such as collisions can occur. Expand
Gradient-based inverse risk-sensitive reinforcement learning
TLDR
This work proposes a gradient-based inverse reinforcement learning algorithm that minimizes a loss function defined on the observed behavior that is compatible with Markov decision processes where the agent is risksensitive. Expand
Inverse Risk-Sensitive Reinforcement Learning
TLDR
A risk-sensitive reinforcement learning algorithm with convergence guarantees that makes use of coherent risk metrics and models of human decision-making which have their origins in behavioral psychology and economics is presented. Expand
Epistemic Risk-Sensitive Reinforcement Learning
TLDR
A framework for interacting with uncertain environments in reinforcement learning by leveraging preferences in the form of utility functions that can be risk-averse, risk-neutral or risk-taking depending on the parameter choice is developed. Expand
Risk-Aware Transfer in Reinforcement Learning using Successor Features
TLDR
This paper addresses the problem of risk-aware policy transfer between tasks in a common domain that differ only in their reward functions, in which risk is measured by the variance of reward streams, and develops risk-awareness features that integrate seamlessly within the RL framework and inherit the superior task generalization ability of SFs. Expand
Risk-Sensitive Reinforcement Learning: A Constrained Optimization Viewpoint
TLDR
This article focuses on the combination of risk criteria and reinforcement learning in a constrained optimization framework, i.e., a setting where the goal to find a policy that optimizes the usual objective of infinite-horizon discounted/average cost, while ensuring that an explicit risk constraint is satisfied. Expand
A Scheme for Dynamic Risk-Sensitive Sequential Decision Making
TLDR
It is shown that most risk measures can be estimated using return variance and, by virtue of the state-augmentation transformation, practical problems modeled by Markov decision processes with stochastic rewards can be solved in a risk-sensitive scenario. Expand
RAIL: Risk-Averse Imitation Learning
TLDR
The proposed RAIL algorithm appears as a potent alternative to GAIL for improved reliability in risk-sensitive applications by minimizing tail risk within the GAIL framework by quantifying tail risk by the Conditional-Value-at-Risk of trajectories and developing the Risk-Averse Imitation Learning (RAIL) algorithm. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 39 REFERENCES
Risk-Sensitive Reinforcement Learning
TLDR
This risk-sensitive reinforcement learning algorithm is based on a very different philosophy and reflects important properties of the classical exponential utility framework, but avoids its serious drawbacks for learning. Expand
Consideration of Risk in Reinforcement Learning
  • M. Heger
  • Mathematics, Computer Science
  • ICML
  • 1994
TLDR
The purpose of this paper is to draw the reader's attention to the problems of the expected value criterion in Markov decision processes and to give Dynamic Programming algorithms for an alternative criterion, namely the minimax criterion. Expand
Neural Prediction Errors Reveal a Risk-Sensitive Reinforcement-Learning Process in the Human Brain
Humans and animals are exquisitely, though idiosyncratically, sensitive to risk or variance in the outcomes of their actions. Economic, psychological, and neural aspects of this are well studied whenExpand
Risk-Sensitive Optimal Feedback Control Accounts for Sensorimotor Behavior under Uncertainty
TLDR
It is shown that subjects change their movement strategy pessimistically in the face of increased uncertainty in accord with the predictions of a risk-averse optimal controller, suggesting that risk-sensitivity is a fundamental attribute that needs to be incorporated into optimal feedback control models. Expand
Human Insula Activation Reflects Risk Prediction Errors As Well As Risk
TLDR
Using functional imaging during a simple gambling task, it is shown that an early-onset activation in the human insula correlates significantly with risk prediction error and that its time course is consistent with a role in rapid updating. Expand
Q-Learning for Risk-Sensitive Control
  • V. Borkar
  • Computer Science
  • Math. Oper. Res.
  • 2002
We propose for risk-sensitive control of finite Markov chains a counterpart of the popular Q-learning algorithm for classical Markov decision processes. The algorithm is shown to converge withExpand
Risk-Sensitivity in Sensorimotor Control
TLDR
Evidence that humans exhibit risk-sensitivity in their motor behaviors, thereby demonstrating sensitivity to the variability of “motor costs,” is reviewed and it is concluded that risk-Sensitivity is an important concept in understanding individual motor behavior under uncertainty. Expand
Advances in prospect theory: Cumulative representation of uncertainty
We develop a new version of prospect theory that employs cumulative rather than separable decision weights and extends the theory in several respects. This version, called cumulative prospect theory,Expand
The economics of risk and time
This book updates and advances the theory of expected utility as applied to risk analysis and financial decision making. Von Neumann and Morgenstern pioneered the use of expected utility theory inExpand
Risk-averse dynamic programming for Markov decision processes
TLDR
The concept of a Markov risk measure is introduced and it is used to formulate risk-averse control problems for two Markov decision models: a finite horizon model and a discounted infinite horizon model. Expand
...
1
2
3
4
...