Recognizing Frustration of Drivers From Face Video Recordings and Brain Activation Measurements With Functional Near-Infrared Spectroscopy

  title={Recognizing Frustration of Drivers From Face Video Recordings and Brain Activation Measurements With Functional Near-Infrared Spectroscopy},
  author={Klas Ihme and Anirudh Unni and Meng Zhang and Jochem W. Rieger and Meike Jipp},
  journal={Frontiers in Human Neuroscience},
Experiencing frustration while driving can harm cognitive processing, result in aggressive behavior and hence negatively influence driving performance and traffic safety. Being able to automatically detect frustration would allow adaptive driver assistance and automation systems to adequately react to a driver’s frustration and mitigate potential negative consequences. To identify reliable and valid indicators of driver’s frustration, we conducted two driving simulator experiments. In the first… 

Facing Driver Frustration: Towards Real-Time In-Vehicle Frustration Estimation Based on Video Streams of the Face

The development of a classifier that can recognize whether a frustrated facial expression was shown based on video streams of the face is described, which can form the basis for the development of frustration-aware vehicles and is foreseen to be extended to more modalities as well as more user need-oriented adaption strategies in the near future.

Protocol for Eliciting Driver Frustration in an In-vehicle Environment

A novel protocol to elicit true frustration of varying intensities in participants interacting with an in-car Human Machine Interface while driving in a simulator is described and baseline machine learning methods trained on features computed from facial and vocal modalities are presented.

Why drivers are frustrated: results from a diary study and focus groups

The main reasons for frustration in driving are related to traffic, in-car reasons, self-inflicted causes, and weather, which will help to design a frustration-aware system that monitors the driver's environment according to the spectrum of frustration causes found in the research presented here.

Towards affect‐aware vehicles for increasing safety and comfort: recognising driver emotions from audio recordings in a realistic driving study

The authors present the automatic analysis of the emotional driver states of frustration, anxiety, positive and neutral, comparable to and even outperforming other reported studies of emotion recognition in the wild.

Shedding light on the prefrontal correlates of mental workload in simulated driving: a functional near-infrared spectroscopy study

This study used functional near-infrared spectroscopy and subjective ratings to measure mental workload in two virtual driving environments with distinct demands and proposed that the right middle frontal gyrus might be a suitable region for the application of powerful small-area brain computer interfaces.

Investigation of Frustration

The measurability of frustration with subjective (questionnaires) and objective (electrocardiogram and electromyogram) methods and the influence on user state factors (like dominance, self-confidence, and performance) in an emotion induction task was examined.

An Experimental Paradigm for Inducing Emotions in a Real World Driving Scenario Evidence from Self-Report, Annotation of Speech Data and Peripheral Physiology

An experimental paradigm is presented that induces four different emotional states in a real-world driving setting using a combination of secondary tasks and conversation-based emotional recall and a list of recommendations for the induction of emotions in real world driving settings is given.

Summarizing Driving Behavior to Support Driver Stress Analysis

The proposed approach for analyzing and summarizing information about driver stress based on their behavior helps a professional to quickly understand the drivers’ profile, interpreting the causes of the Drivers’ reactions, thus providing more accurate assistance.

A Methodological Review of fNIRS in Driving Research: Relevance to the Future of Autonomous Vehicles

A methodological review of the existing fNIRS driving research is provided, arguing that standardization of these approaches will facilitate greater overlap of methods by researchers from all disciplines, which will allow for meta-analysis of future results.



Frustration in the face of the driver

Recording facial muscle behavior potentially provides traffic researchers and assistance system developers with the possibility to recognize frustration while driving.

Analysis of Real-World Driver's Frustration

This paper investigates a method for estimating a driver's spontaneous frustration in the real world using a Bayesian network, and combines knowledge on the driving environment assessed through data annotation, speech recognition errors, the driver's emotional state (frustration), and theDriver's responses measured through facial expressions, physiological condition, and gas- and brake-pedal actuation.

Automatically Recognizing Facial Indicators of Frustration: A Learning-centric Analysis

A study to analyze a video corpus of computer-mediated human tutoring using an automated facial expression recognition tool that detects fine-grained facial movements reveals three significant relationships between facial expression, frustration, and learning.

Detecting emotional stress from facial expressions for driving safety

A real-time non-intrusive monitoring system is developed, which detects the emotional states of the driver by analyzing facial expressions, and which operates very well on simulated data even with generic models.

Measuring Drivers' Frustration in a Driving Simulator

The objective of the current study was to measure the effect of frustration on drivers' performance and scanning efficiency in a driving simulator. We hypothesized that unsafe driving would be

Assessing the Driver’s Current Level of Working Memory Load with High Density Functional Near-infrared Spectroscopy: A Realistic Driving Simulator Study

W whole head fNIRS sampling revealed increasing brain activation in bilateral inferior frontal and bilateral temporo-occipital brain areas with increasing working memory load levels suggesting that these areas are specifically involved in workload-related processing.

Using Noninvasive Brain Measurement to Explore the Psychological Effects of Computer Malfunctions on Users during Human-Computer Interactions

An experiment was conducted where participants conducted a series of web search tasks while wearing functional near-infrared spectroscopy (fNIRS) and galvanic skin response sensors, suggesting that fNIRs can be used to measure the different cognitive and emotional responses associated with computer malfunctions.

Exploring Temporal Patterns in Classifying Frustrated and Delighted Smiles

An automated system is developed to distinguish between naturally occurring spontaneous smiles under frustrating and delightful stimuli by exploring their temporal patterns given video of both and extracting local and global features related to human smile dynamics.