Addressing Inherent Uncertainty: Risk-Sensitive Behavior Generation for Automated Driving using Distributional Reinforcement Learning

@article{Bernhard2019AddressingIU,
  title={Addressing Inherent Uncertainty: Risk-Sensitive Behavior Generation for Automated Driving using Distributional Reinforcement Learning},
  author={Julian Bernhard and Stefan Pollok and Alois Knoll},
  journal={2019 IEEE Intelligent Vehicles Symposium (IV)},
  year={2019},
  pages={2148-2155}
}
For highly automated driving above SAE level 3, behavior generation algorithms must reliably consider the inherent uncertainties of the traffic environment, e.g. arising from the variety of human driving styles. Such uncertainties can generate ambiguous decisions, requiring the algorithm to appropriately balance low-probability hazardous events, e.g. collisions, and high-probability beneficial events, e.g. quickly crossing the intersection. State-of-the-art behavior generation algorithms lack a… 

Figures and Tables from this paper

Risk-Constrained Interactive Safety under Behavior Uncertainty for Autonomous Driving

This work formalizes this safety objective, presents the Risk-Constrained Robust Stochastic Bayesian Game modeling interactive decisions satisfying a maximum risk of violating a safety envelope under uncertainty of other traffic participants' behavior and solves it using the variant of Multi-Agent Monte Carlo Tree Search.

Ensemble Quantile Networks: Uncertainty-Aware Reinforcement Learning with Applications in Autonomous Driving

The results show that the EQN method can balance risk and time efficiency in different occluded intersection scenarios, by considering the estimated aleatoric uncertainty, and it is shown that the trained agent can use the epistemic uncertainty information to identify situations that the agent has not been trained for and thereby avoid making unfounded, potentially dangerous, decisions outside of the training distribution.

Uncertainty-aware Energy Management of Extended Range Electric Delivery Vehicles with Bayesian Ensemble

The method of Bayesian ensemble is adapted to train a group of agents with imposed diversity for an energy management system of a delivery vehicle and the agents in the ensemble agree well on familiar states but show diverse results on unfamiliar or novel states.

Cooperative Trajectory Planning in Uncertain Environments with Monte Carlo Tree Search and Risk Metrics

This work extends an existing cooperative trajectory planning approach based on Monte Carlo Tree Search for continuous action spaces by explicitly modeling uncertainties in the form of a root belief state, from which start states for trees are sampled.

Safer Autonomous Driving in a Stochastic, Partially-Observable Environment by Hierarchical Contingency Planning

An end-to-end approach, from training to execution, for learning robust contingency plans and combining them with a hierarchical planner to obtain a robust agent policy in an autonomous navigation task where other vehicles’ behaviours are unknown, and the agent’s belief about these behaviours is subject to sudden, last-second change.

Graph Convolution-Based Deep Reinforcement Learning for Multi-Agent Decision-Making in Mixed Traffic Environments

This research proposes a framework to enable different Graph Reinforcement Learning (GRL) methods for decision-making, and compares their performance in interactive driving scenarios, and shows that the implementation of GNN can well represents the interaction between vehicles, and the combination of DRL and GNN is able to improve the performance of the generation of lanechange behaviors.

Mean-variance Based Risk-sensitive Reinforcement Learning with Interpretable Attention

An interpretable visualization scheme with attention mechanism in which a saliency map represents the relative influence degree of an input state on the decision-making of mean-variance based risk-sensitive RL.

A Survey of Deep RL and IL for Autonomous Driving Policy Learning

This is the first survey to focus on AD policy learning using DRL/DIL, which is addressed simultaneously from the system, task-driven and problem-driven perspectives.

S AFER A UTONOMOUS D RIVING IN A S TOCHASTIC , P ARTIALLY -O BSERVABLE E NVIRONMENT BY H IERAR CHICAL C ONTINGENCY P LANNING

An end-to-end approach, from training to execution, for learning robust contingency plans and combining them with a hierarchical planner to obtain a robust agent policy in an autonomous navigation task where other vehicles’ behaviours are unknown, and the agent’s belief about these behaviours is subject to sudden, last-second change.

Decision-Making Technology for Autonomous Vehicles: Learning-Based Methods, Applications and Future Outlook

A brief review on learning-based decision-making technology for autonomous vehicles since it is significant for safer and efficient performance of autonomous vehicles.

References

SHOWING 1-10 OF 30 REFERENCES

Risk-sensitive Inverse Reinforcement Learning via Coherent Risk Models

Comparisons of the RiskSensitive (RS) IRL approach with a risk-neutral model show that the RS-IRL framework more accurately captures observed participant behavior both qualitatively and quantitatively.

Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving

This paper applies deep reinforcement learning to the problem of forming long term driving strategies and shows how policy gradient iterations can be used without Markovian assumptions, and decomposes the problem into a composition of a Policy for Desires and trajectory planning with hard constraints.

High-level Decision Making for Safe and Reasonable Autonomous Lane Changing using Reinforcement Learning

A deep reinforcement learning (RL) agent is let to drive as close as possible to a desired velocity by executing reasonable lane changes on simulated highways with an arbitrary number of lanes by making use of a minimal state representation and a Deep Q-Network.

Reinforcement Learning with Probabilistic Guarantees for Autonomous Driving

This paper outlines a case study of an intersection scenario involving multiple traffic participants and proposes a generic approach to enforce probabilistic guarantees on an RL agent that outperforms a rule-based heuristic approach in terms of efficiency while exhibiting strong guarantees on safety.

Deep Reinforcement Learning with Risk-Seeking Exploration

This paper proposes a novel DRL algorithm that encourages risk-seeking behaviour to enhance information acquisition during training and demonstrates the merit of the exploration heuristic by arguing that the risk estimator implicitly contains both parametric uncertainty and inherent uncertainty of the environment.

Adaptive Behavior Generation for Autonomous Driving using Deep Reinforcement Learning with Compact Semantic States

Deep Reinforcement Learning is used to learn maneuver decisions based on a compact semantic state representation, ensuring a consistent model of the environment across scenarios as well as a behavior adaptation function, enabling on-line changes of desired behaviors without re-training.

Tactical Decision Making for Lane Changing with Deep Reinforcement Learning

This paper presents a framework that demonstrates a more structured and data efficient alternative to end-to-end complete policy learning on problems where the high-level policy is hard to formulate using traditional optimization or rule based methods but well designed low-level controllers are available.

Experience-Based Heuristic Search: Robust Motion Planning with Deep Q-Learning

The Experience-Based-Heuristic-Search algorithm is proposed, which overcomes the statistical failure rate of a Deep-reinforcement-Iearning-based planner and still benefits computationally from the pre-learned optimal policy.

A Belief State Planner for Interactive Merge Maneuvers in Congested Traffic

A novel motion model representing the uncertain cooperation of other drivers is presented based on a logistic regression model estimating the probability for cooperative behavior of a human driver given a future scene.

Distributional Reinforcement Learning with Quantile Regression

This paper examines methods of learning the value distribution instead of the value function in reinforcement learning, and presents a novel distributional reinforcement learning algorithm consistent with the theoretical formulation.