STEP: Spatial Temporal Graph Convolutional Networks for Emotion Perception from Gaits

@inproceedings{Bhattacharya2019STEPST,
  title={STEP: Spatial Temporal Graph Convolutional Networks for Emotion Perception from Gaits},
  author={Uttaran Bhattacharya and Trisha Mittal and Rohan Chandra and Tanmay Randhavane and Aniket Bera and Dinesh Manocha},
  booktitle={AAAI Conference on Artificial Intelligence},
  year={2019}
}
We present a novel classifier network called STEP, to classify perceived human emotion from gaits, based on a Spatial Temporal Graph Convolutional Network (ST-GCN) architecture. Given an RGB video of an individual walking, our formulation implicitly exploits the gait features to classify the perceived emotion of the human into one of four emotions: happy, sad, angry, or neutral. We train STEP on annotated real-world gait videos, augmented with annotated synthetic gaits generated using a novel… 

Figures and Tables from this paper

Learning Gait Emotions Using Affective and Deep Features

A novel data-driven algorithm to learn the perceived emotions of individuals based on their walking motion or gaits, which identifies both the categorical emotions from the gaits and the corresponding values for the dimensional emotion components - valence and arousal.

Take an Emotion Walk: Perceiving Emotions from Gaits Using Hierarchical Attention Pooling and Affective Mapping

An autoencoder-based semi-supervised approach to classify perceived human emotions from walking styles obtained from videos or motion-captured data and represented as sequences of 3D poses outperform current state-of-art algorithms for both emotion recognition and action recognition from 3D gaits by 7%--23% on the absolute.

MSA-GCN: Multiscale Adaptive Graph Convolution Network for Gait Emotion Recognition

A novel MultiScale Adaptive Graph Convolution Network (MSA- GCN) is presented, designed to select the convolution kernel dynamically to obtain the soft spatio- temporal features of different emotions.

TNTC: Two-Stream Network with Transformer-Based Complementarity for Gait-Based Emotion Recognition

A novel two-stream network with transformer-based complementarity, termed as TNTC, which outperforms state-of-the-art methods on the latest dataset in terms of accuracy and bridge the complementarity between two streams hierarchically via capturing long range dependencies.

Motion Capture Sensor-Based Emotion Recognition Using a Bi-Modular Sequential Neural Network

A new lightweight bi-modular architecture with handcrafted features that is trained using a RMSprop optimizer and stratified data shuffling is proposed, which outperforms all recent deep-learning methods and is highly effective in correctly inferring human emotions from gait.

Dynamic Emotion Modeling With Learnable Graphs and Graph Inception Network

The Learnable Graph Inception Network (L-GrIN) is presented, that jointly learns to recognize emotion and to identify the underlying graph structure in the dynamic data by modeling dynamic data as structured graphs.

ProxEmo: Gait-based Emotion Learning and Multi-view Proxemic Fusion for Socially-Aware Robot Navigation

This work presents ProxEmo, a novel end-to-end emotion prediction algorithm for socially aware robot navigation among pedestrians that outperform current state-of-art algorithms for emotion recognition from 3D gaits.

A LSTM-based Approach for Gait Emotion Recognition

This paper contributes to the domain of knowledge by presenting a novel light architecture for inferring human emotions through gait that outperforms all recent deep learning methods, while having the lowest inference time for each gait sample.

Learnable Graph Inception Network for Emotion Recognition

This work proposes a generalized graph approach to emotion recognition that can take any time-varying data modality as input and achieves state-of-the-art performance on all databases outperforming several competitive baselines and relevant existing methods.

Leveraging Semantic Scene Characteristics and Multi-Stream Convolutional Architectures in a Contextual Approach for Video-Based Visual Emotion Recognition in the Wild

This work tackles the task of video-based visual emotion recognition in the wild by leveraging visual context in the form of scene characteristics and attributes, as part of a broader emotion recognition framework and surpasses the previous best published recognition scores by a large margin.
...

References

SHOWING 1-10 OF 95 REFERENCES

Take an Emotion Walk: Perceiving Emotions from Gaits Using Hierarchical Attention Pooling and Affective Mapping

An autoencoder-based semi-supervised approach to classify perceived human emotions from walking styles obtained from videos or motion-captured data and represented as sequences of 3D poses outperform current state-of-art algorithms for both emotion recognition and action recognition from 3D gaits by 7%--23% on the absolute.

Identifying Emotions from Walking using Affective and Deep Features

This work presents a new data-driven model and algorithm to identify perceived emotions of individuals based on their walking styles and presents an EWalk (Emotion Walk) dataset that consists of videos of walking individuals with gaits and labeled emotions.

Adaptive Real-Time Emotion Recognition from Body Movements

A real-time system that continuously recognizes emotions from body movements is proposed, and a novel semisupervised adaptive algorithm is built on top of the conventional Random Forests classifier.

Emotion Recognition through Gait on Mobile Devices

  • Mangtik ChiuJiayu ShuP. Hui
  • Computer Science
    2018 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops)
  • 2018
This paper explores the possibility of emotion recognition through gait, which is one of the most common human behaviors, and trains several supervised learning models, including SVM, Multilayer Perceptron, Naive Bayes, Decision Tree, Random Forest and Logistic Regression.

Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition

A novel model of dynamic skeletons called Spatial-Temporal Graph Convolutional Networks (ST-GCN), which moves beyond the limitations of previous methods by automatically learning both the spatial and temporal patterns from data.

Emotion Recognition by Body Movement Representation on the Manifold of Symmetric Positive Definite Matrices

This work uses sequences of spontaneous walking under the five primary emotional states to report a method that succeeded in classifying the different emotions, with comparable performance to those observed in a human-based force-choice classification task.

Spatiotemporal Residual Networks for Video Action Recognition

The novel spatiotemporal ResNet is introduced and evaluated using two widely used action recognition benchmarks where it exceeds the previous state-of-the-art.

3D Convolutional Neural Networks for Human Action Recognition

A novel 3D CNN model for action recognition that extracts features from both the spatial and the temporal dimensions by performing 3D convolutions, thereby capturing the motion information encoded in multiple adjacent frames.

Temporal Convolutional Networks for Action Segmentation and Detection

A class of temporal models that use a hierarchy of temporal convolutions to perform fine-grained action segmentation or detection, which are capable of capturing action compositions, segment durations, and long-range dependencies, and are over a magnitude faster to train than competing LSTM-based Recurrent Neural Networks.

Recognizing Emotions Conveyed by Human Gait

By using the gait characteristics it is possible to characterize each emotion with good accuracy for intra-subject data-base, and for inter-subject database results show that recognition is more prone to error, suggesting strong inter-personal differences in emotional features.
...