Convolutional MKL Based Multimodal Emotion Recognition and Sentiment Analysis

@article{Poria2016ConvolutionalMB,
  title={Convolutional MKL Based Multimodal Emotion Recognition and Sentiment Analysis},
  author={Soujanya Poria and Iti Chaturvedi and E. Cambria and Amir Hussain},
  journal={2016 IEEE 16th International Conference on Data Mining (ICDM)},
  year={2016},
  pages={439-448}
}
Technology has enabled anyone with an Internet connection to easily create and share their ideas, opinions and content with millions of other people around the world. Much of the content being posted and consumed online is multimodal. With billions of phones, tablets and PCs shipping today with built-in cameras and a host of new video-equipped wearables like Google Glass on the horizon, the amount of video on the Internet will only continue to increase. It has become increasingly difficult for… 

Figures and Tables from this paper

Multimodal sentiment analysis in social media using deep learning with convolutional neural networks
TLDR
This thesis deals with the sentiment analysis of videos, where information is available in three modalities: audio, video, and text, hence multimodal sentiment analysis, and presents two novel methods of modality fusion: fusing in a hierarchical fashion using perceptrons and employing attention mechanism.
Ensemble application of ELM and GPU for real-time multimodal sentiment analysis
TLDR
This work leveraged on the processing speed of extreme learning machine and graphics processing unit to overcome the limitations of standard learning algorithms and central processing unit (CPU) and perform real-time multimodal sentiment analysis, i.e., harvesting sentiments from web videos by taking into account audio, visual and textual modalities as sources of the information.
Deep Neural Networks for Emotion Recognition Combining Audio and Transcripts
TLDR
Fusing audio and transcript information improved unweighted accuracy by relative 24% for IEMOCAP and relative 3.4% for the telephone data compared to a single acoustic system.
Visual and Textual Sentiment Analysis of Daily News Social Media Images by Deep Learning
TLDR
This paper proposes an innovative approach for analyzing both visual and textual features of Social Media images using Deep Convolutional Neural Networks (DCNNs), in order to collect more accurate results than the single analysis of both type can do alone.
A Deep Multi-Level Attentive network for Multimodal Sentiment Analysis
TLDR
A Deep Multi-level Attentive network (DMLANet), which exploits the correlation between image and text modalities to improve multimodal learning and conducts extensive evaluations on four real-world datasets, which verify the method's superiority.
Learning Visual Concepts in Images Using Temporal Convolutional Networks
TLDR
This paper proposes to automatically predict the corresponding sentiments using deep convolutional networks and considers a recurrent neural network to model the sequence of different sentiments in a single image.
An RNN-Based Multimodal Sentiment Analysis: Focusing on Facial Expressions and Multimodal Dynamic Representations
  • Computer Science
  • 2018
TLDR
A multimodal opinion classi€cation system fully constructed on deep learning technology, thus taking into account the time-dependency of the data, that outperforms a CNN-based system on the linguistic modality of the MOUD dataset.
Sentiment analysis in textual, visual and multimodal inputs using recurrent neural networks
TLDR
An extensive review over the applicability, challenges, issues, and approaches over the role of sequential deep neural networks in sentiment analysis of multimodal data using RNN and its architectural variants is presented.
A Deep Neural Network Model for the Detection and Classification of Emotions from Textual Content
TLDR
A deep learning model is presented, bidirectional long-term short-term memory (BiLSMT), for emotion recognition that takes into account five main emotions, which helps firms to apply best practices in the selection, management, and optimization of policies, services, and product information.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 38 REFERENCES
Towards multimodal sentiment analysis: harvesting opinions from the web
TLDR
This paper addresses the task of multimodal sentiment analysis, and conducts proof-of-concept experiments that demonstrate that a joint model that integrates visual, audio, and textual features can be effectively used to identify sentiment in Web videos.
Utterance-Level Multimodal Sentiment Analysis
TLDR
It is shown that multimodal sentiment analysis can be effectively performed, and that the joint use of visual, acoustic, and linguistic modalities can lead to error rate reductions of up to 10.5% as compared to the best performing individual modality.
Deep Convolutional Neural Network Textual Features and Multiple Kernel Learning for Utterance-level Multimodal Sentiment Analysis
TLDR
A novel way of extracting features from short texts, based on the activation values of an inner layer of a deep convolutional neural network, is presented and a parallelizable decision-level data fusion method is presented, which is much faster, though slightly less accurate.
Towards an intelligent framework for multimodal affective data analysis
Affective Computing and Sentiment Analysis
  • E. Cambria
  • Computer Science
    IEEE Intelligent Systems
  • 2016
TLDR
The emerging fields of affective computing and sentiment analysis, which leverage human-computer interaction, information retrieval, and multimodal signal processing for distilling people's sentiments from the ever-growing amount of online social data.
A Deeper Look into Sarcastic Tweets Using Deep Convolutional Neural Networks
TLDR
This work develops models based on a pre-trained convolutional neural network for extracting sentiment, emotion and personality features for sarcasm detection and addresses the often ignored generalizability issue of classifying data that have not been seen by the models at learning phase.
Multimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal Messages
TLDR
This article addresses the fundamental question of exploiting the dynamics between visual gestures and verbal messages to be able to better model sentiment by introducing the first multimodal dataset with opinion-level sentiment intensity annotations and proposing a new computational representation, called multi-modal dictionary, based on a language-gesture study.
Aspect extraction for opinion mining with a deep convolutional neural network
Ensemble of SVM trees for multimodal emotion recognition
TLDR
The experiments show that the proposed ensemble of trees of binary SVM classifiers outperforms classical multi-way SVM classification with one-vs-one voting scheme and achieves state-of-the-art results for all feature combinations.
...
1
2
3
4
...