Simultaneous policy learning and latent state inference for imitating driver behavior

@article{Morton2017SimultaneousPL,
  title={Simultaneous policy learning and latent state inference for imitating driver behavior},
  author={Jeremy Morton and Mykel J. Kochenderfer},
  journal={2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC)},
  year={2017},
  pages={1-6}
}
  • Jeremy MortonMykel J. Kochenderfer
  • Published 19 April 2017
  • Computer Science, Psychology
  • 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC)
Human driving depends on latent states, such as aggression and intent, that cannot be directly observed. In this work, we propose a method for learning driver models that can account for unobserved states. When trained on a synthetic dataset, our model is able to learn encodings for vehicle trajectories that distinguish between four distinct classes of driver behavior. Such encodings are learned without any knowledge of the number of driver classes or any objective that directly requires the… 
View on IEEE
arxiv.org
27 Citations
Highly Influential Citations
4
Background Citations
19
Methods Citations
9
Results Citations
1

Figures and Tables from this paper

27 Citations

Reinforcement Learning for Autonomous Driving with Latent State Inference and Spatial-Temporal Relationships

This work encodes prior knowledge on the latent states of other drivers through a framework that combines the reinforcement learner with a supervised learner and model the influence passing between different vehicles through graph neural networks (GNNs).

Learning to Navigate Intersections with Unsupervised Driver Trait Inference

This work proposes an unsupervised method for inferring driver traits such as driving styles from observed vehicle trajectories using a variational autoencoder with recurrent neural networks and uses this trait representation to learn a policy for an autonomous vehicle to navigate through a T-intersection with deep reinforcement learning.

Learning Latent Traits for Simulated Cooperative Driving Tasks

This paper builds a framework capable of capturing a compact latent representation of the human in terms of their behavior and preferences based on data from a simulated population of drivers, using a lightweight simulation environment for modelling one form of distracted driving behavior.

Learning a Multi-Modal Policy via Imitating Demonstrations with Mixed Behaviors

This work proposes a novel approach to train a multi-modal policy from mixed demonstrations without their behavior labels based on the variational autoencoder with a categorical latent variable, and develops a method to discover the latent factors of variation in the demonstrations.

Learning an Interpretable Model for Driver Behavior Prediction with Inductive Biases

This work embeds the Intelligent Driver Model, a popular hand-crafted driver model, into deep neural networks, and yields a robust model that is end-to-end trainable and provides greater transparency at no cost to the model's predictive accuracy.

Modeling Human Driving Behavior Through Generative Adversarial Imitation Learning

Experiments show that modifications to GAIL can successfully model highway driving behavior, accurately replicating human demonstrations and generating realistic, emergent behavior in the traffic flow arising from the interaction between driving agents.

Learning Temporal Strategic Relationships using Generative Adversarial Imitation Learning

A novel framework for automatic learning of complex strategies in human decision making, which uses two external memory modules for capturing dependencies within a single expert demonstration, and a global memory module for modelling task level characteristics such as best practice employed by different humans based on their domain expertise.

Direct Policy Transfer via Hidden Parameter Markov Decision Processes

Many situations arise in which an agent must learn to solve tasks with similar, but not identical, dynamics. For example, a robot might be tasked to manipulate objects with slightly different masses…

Burn-In Demonstrations for Multi-Modal Imitation Learning

This work extends InfoGAIL, an algorithm for multi-modal imitation learning, to reproduce behavior over an extended period of time, and involves reformulating the typical imitation learning setting to include "burn-in demonstrations" upon which policies are conditioned at test time.

HMIway-env: A Framework for Simulating Behaviors and Preferences to Support Human-AI Teaming in Driving

A lightweight simulation and modeling framework, HMIway-env, for studying human-machine teaming in the context of driving is introduced, incorporating models for distracted and cautious driving, and early experimental results toward the training of better intervention policies are shown.

References

SHOWING 1-10 OF 22 REFERENCES

Imitating driver behavior with generative adversarial networks

This paper adopts a method for overcoming the problem of cascading errors inherent in prior approaches, resulting in realistic behavior that is robust to trajectory perturbations, and extends Generative Adversarial Imitation Learning to the training of recurrent policies.

High-speed highway scene prediction based on driver models learned from demonstrations

This work shows how to use a realistic driver model learned from demonstrations via Inverse Reinforcement Learning to predict the long-term evolution of highway traffic scenes and demonstrates the capability of this approach to produce reliable, human-like scene predictions.

Analysis of Recurrent Neural Networks for Probabilistic Modeling of Driver Behavior

This paper reveals that the strong performance of recurrent networks is due to the ability of the recurrent network to identify recent trends in the ego-vehicle's state, and recurrent networks are shown to perform as, well as feedforward networks with longer histories as inputs.

Characterizing Driving Styles with Deep Learning

A novel deep learning solution toCharacterizing driving styles of human drivers using vehicle sensor data, e.g., GPS, is proposed, which could be the first attempt of extending deep learning to driving behavior analysis based on GPS data.

The value of inferring the internal state of traffic participants for autonomous freeway driving

This research uses a simple model for human behavior with unknown parameters that make up the internal states of the traffic participants and presents a method for quantifying the value of estimating these states and planning with their uncertainty explicitly modeled.

Generative Adversarial Imitation Learning

A new general framework for directly extracting a policy from data, as if it were obtained by reinforcement learning following inverse reinforcement learning, is proposed and a certain instantiation of this framework draws an analogy between imitation learning and generative adversarial networks.

Efficient Reductions for Imitation Learning

This work proposes two alternative algorithms for imitation learning where training occurs over several episodes of interaction and shows that this leads to stronger performance guarantees and improved performance on two challenging problems: training a learner to play a 3D racing game and Mario Bros.

Model-Free Imitation Learning with Policy Optimization

Under the apprenticeship learning formalism, this work develops alternative model-free algorithms for finding a parameterized stochastic policy that performs at least as well as an expert policy on an unknown cost function, based on sample trajectories from the expert.

End to End Learning for Self-Driving Cars

A convolutional neural network is trained to map raw pixels from a single front-facing camera directly to steering commands and it is argued that this will eventually lead to better performance and smaller systems.

Analysis of microscopic behavior models for probabilistic modeling of driver behavior

A comparative evaluation of several probabilistic microscopic human behavior models from the literature trained on naturalistic data for free-flow, car following, and lane change context-classes on highways shows that a new class of Bayesian network models outperforms the state of the art.