DMER

A survey of deep multimodal emotion recognition.

Performance Comparsion

Summary of latest papers

Update

2022.03.21 Add papers from ACM MM 2021

2022.05.04 Add the pages of performance comparsion and the summary of latest papers.

---

Structure

• Related Github Repositories
• Datasets
• Related Challenges
• Related Projects
• Related Reviews
• Related Papers

Related Github Repositories

• AWESOME-MER
• AWESOME-MSA
• SER-datasets
• Awesome-Speech-Pretraining
• Reading list for Awesome Sentiment Analysis papers
• Speech-Resources

Datasets

Datasets	Year	Features	Paper	Used
MEmoR	ACM MM 2020	Visual, Audio, Text transcripts	Memor: A dataset for multimodal emotion reasoning in videos
EMOTIC Dataset	2019 TPAMI	Face, Context	Context based emotion recognition using emotic dataset
CMU-MOSEI	ACL 2018	Visual, Audio, Language	Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph
ASCERTAIN Dataset	2018 TAC	Facial activity data, Physiological data	ASCERTAIN: Emotion and Personality Recognition Using Commercial Sensors
RAVDESS	2018 PLoS ONE	Visual, Audio	The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English
CMU-MOSI	2016 IEEE Intelligent Systems	Visual, Audio, Language	Multimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal Messages
Multimodal Spontaneous Emotion Database (BP4D+)	CVPR 2016	Face, Thermal data, Physiological data	Multimodal Spontaneous Emotion Corpus for Human Behavior Analysis
EmotiW Database		Visual, Audio
LIRIS-ACCEDE Database	2015 TAC	Visual, Audio	LIRIS-ACCEDE: A Video Database for Affective Content Analysis
CREMA-D	2014 TAC	Visual, Audio	CREMA-D: Crowd-Sourced Emotional Multimodal Actors Dataset
POM	ICMI 2014	Acoustic Descriptors, Verbal and Para-Verbal Descriptors, Visual Descriptors	Computational Analysis of Persuasiveness in Social Multimedia: A Novel Dataset and Multimodal Prediction Approach
SEMAINE Database	2014 TAC	Visual, Audio	CREMA-D: Crowd-Sourced Emotional Multimodal Actors Dataset
MAHNOB-HCI	2011 TAC	Visual, Eye gaze, Physiological data	A Multimodal Database for Affect Recognition and Implicit Tagging
IEMOCAP Database	2008 LRE	Visual, Audio, Text transcripts	IEMOCAP: Interactive emotional dyadic motion capture database
eNTERFACE Dataset	ICDEW 2006	Visual, Audio	The eNTERFACE'05 audio-visual emotion database

Related Challenges

• Multimodal (Audio, Facial and Gesture) based Emotion Recognition Challenge (MMER) @ FG
• Emotion Recognition in the Wild Challenge (EmotiW) @ ICMI
• Audio/Visual Emotion Challenge (AVEC) @ ACM MM
• One-Minute Gradual-Emotion Behavior Challenge @ IJCNN
• Multimodal Emotion Recognition Challenge (MEC) @ ACII
• Multimodal Pain Recognition (Face and Body) Challenge (EmoPain) @ FG

Related Projects

• CMU Multimodal SDK
• DELTA
• SenticNet

Related Reviews

• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications--(IEEE Journal of Selected Topics in Signal Processing, 2020)
• A snapshot research and implementation of multimodal information fusion for data-driven emotion recognition--(Information Fusion, 2020)
• Survey on AI-Based Multimodal Methods for Emotion Detection--(High-Performance Modelling and Simulation for Big Data Applications, 2019)
• Multimodal machine learning: A survey and taxonomy--(IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018)
• A review of affective computing: From unimodal analysis to multimodal fusion--(Information Fusion, 2017)
• A survey of multimodal sentiment analysis--(Image and Vision Computing, 2017)
• A Review and Meta-Analysis of Multimodal Affect Detection Systems--(ACM Computing Surveys, 2015)

Video-Audio Method

Index	Model	Paper	Year	Project	Dataset	Method
VA-0	VAANet	An End-to-End Visual-Audio Attention Network for Emotion Recognition in User-Generated Videos	AAAI 2020	[coding]	VideoEmotion-8, Ekman-6	Visual+Audio, Attention-Based Model
VA-1		Audio-Visual Emotion Forecasting: Characterizing and Predicting Future Emotion Using Deep Learning	FG 2019	[coding]	IEMOCAP	Face + Speech, Emotion forecasting
VA-2	MMDDN	Multimodal Deep Denoise Framework for Affective Video Content Analysi	MM 2019		LIRIS-ACCEDE	Visual(colors, facial expressions, human gestures) + Audio(pitch, tone, and background music)
VA-3	DBN	Deep learning for robust feature generation in audiovisual emotion recognition	ICASSP 2019		IEMOCAP	Visual + Audio
VA-4	MAFN	Multi-Attention Fusion Network for Video-based Emotion Recognition	ICMI 2019		AFEW	Visual+Audio, Multiple attention fusion network
VA-5	MERML	Metric Learning-Based Multimodal Audio-Visual Emotion Recognition	2019 TMM		eNTERFACE, CREMA-D	Visual+Audio, Video and Audio Feature Extraction and Aggregation, MERML
VA-6		Audio-Visual Emotion Recognition in Video Clips	2019 TAC		SAVEE, NTERFACE’05, RML	Visual + Audio
VA-7	EmoBed	EmoBed: Strengthening Monomodal Emotion Recognition via Training with Crossmodal Emotion Embeddings	2019 TAC		s RECOLA, OMG-Emotion	Face + Audio
VA-8		Affective video content analysis based on multimodal data fusion in heterogeneous networks	2019 Information Fusion			Visual + Audio
VA-9		Audio-visual emotion fusion (AVEF): A deep efficient weighted approach	2019 Information Fusion		RML, Enterface05, BAUM-1s	Visual + Audio
VA-10		Joint low rank embedded multiple features learning for audio–visual emotion recognition	2019 Neurocomputing			Visual + Audio
VA-11	MLRF	Multimodal Local-Global Ranking Fusion for Emotion Recognition	ICMI 2018		AVEC16(RECOLA)	Visual + Audio
VA-12		A Combined Rule-Based & Machine Learning Audio-Visual Emotion Recognition Approach	2018 TAC		Cohn-Kanade, TheENTERFACE’05	Visual + Audio
VA-13	MMDRBN	A Multimodal Deep Regression Bayesian Network for Affective Video Content Analyses	ICCV 2017		LIRIS-ACCEDE	Video + Audio, Multimodal deep regression Bayesian network
VA-14	ASER	Automatic speech emotion recognition using recurrent neural networks with local attention	ICASSP 2017	[coding]	IEMOCAP	Speech
VA-15		Modeling Multimodal Cues in a Deep Learning-Based Framework for Emotion Recognition in the Wild	ICMI 2017		AFEW, FER-2013, eNTERFACE dataset	Visual + Audio
VA-16		Emotion recognition with multimodal features and temporal models	ICMI 2017		EmotiW(2017)	Visual+Audio, Temporal model(LSTM) for facial features
VA-17		Learning Affective Features With a Hybrid Deep Model for Audio–Visual Emotion Recognition	2017 T-CSVT	[coding]	RML dataset, eNTERFACE05, BAUM-1s	Visual(3DCNN) +(DBN) Audio(CNN)
VA-18	EEMER	End-to-End Multimodal Emotion Recognition Using Deep Neural Networks	2017 IEEE Journal of Selected Topics in Signal Processing	[coding]	RECOLA	Visual + Audio
VA-19	MCEF	Multi-cue fusion for emotion recognition in the wild	ICMI 2016		cohn-kanade+, enterface’05	Visual + Audio
VA-20	DemoFBVP	Multimodal emotion recognition using deep learning architectures	WACV 2016		[emoFBVP]	Visual + Audio
VA-21		Exploring Cross-Modality Affective Reactions for Audiovisual Emotion Recognition	2013 TAC		IEMOCAP, SEMAINE	Face + Audio
VA-22		Kernel Cross-Modal Factor Analysis for Information Fusion With Application to Bimodal Emotion Recognition	2012 TMM		RML, eNTERFACE	Visual + Audio, Kernel cross-modal factor analysis

Context-awarded method

Index	Model	Paper	Year	Project	Dataset	Method
CA-1	EmotiCon	EmotiCon: Context-Aware Multimodal Emotion Recognition using Frege’s Principle	CVPR 2020	[video] [project]	EMOTIC, [GroupWalk]	Face+Gait+(Depth+Background), Multiplicative fusion, etc
CA-2	CAER-Net	Context-Aware Emotion Recognition Networks	ICCV 2019	[coding][project]	EMOTIC, AffectNet, [CAER-S], AFEW, [CAER]	Face + Context, Adaptive Fusion
CA-3		Context-aware affective graph reasoning for emotion recognition	ICME 2019
CA-4		Context Based Emotion Recognition using EMOTIC Dataset	2019 TPAMI	[coding]	EMOTIC	Face + Context
CA-5		Multimodal Framework for Analyzing the Affect of a Group of People	2018 TMM		HAPPEI, GAFF	Face+Upper body+Scene, Face-based Group-level Emotion Recognition
CA-6		Emotion Recognition in Context	CVPR 2017	[project]	[EMOTIC]	Body feature+Image feature(Context)

Video-Audio-Text method

Index	Model	Paper	Year	Project	Dataset	Method
VAT-1	Self-MM	Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis	AAAI 2021	[coding]	CMU-MOSI, CMU-MOSEI, SIMS	Video+Speech+Text
VAT-2	CTNet	CTNet: Conversational Transformer Network for Emotion Recognition	2021 TASLP	-	IEMOCAP, MELD	Face+Speech+Text
VAT-3	M3ER	M3ER: Multiplicative Multimodal Emotion Recognition Using Facial, Textual, and Speech Cues	AAAI 2020	[video] [project]	IEMOCAP, CMU-MOSEI	Face+Speech+Text, Multiplicative Fusion, CCA, Modality Check
VAT-4	ARGF	Modality to modality translation: An adversarial representation learning and graph fusion network for multimodal fusion	AAAI 2020		IEMOCAP, CMU-MOSEI, CMU-MOSI	Face+Speech+Text
VAT-5	ICCN	Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis	AAAI 2020		IEMOCAP, CMU-MOSI, CMU-MOSEI	Video+Audio+Text
VAT-6	MISA	MISA: Modality-Invariant and-Specific Representations for Multimodal Sentiment Analysis	ACM MM 2020	[coding]	MOSI and MOSEI	Face+Speech+Text,
VAT-7	CM-BERT	CM-BERT: Cross-Modal BERT for Text-Audio Sentiment Analysis	ACM MM 2020	[coding]	MOSI and MOSEI	Speech+Text
VAT-8	SWAFN	SWAFN: Sentimental Words Aware Fusion Network for Multimodal Sentiment Analysis	ACL 2020	[coding1] [coding2]	CMU-MOSI, CMU-MOSEI and YouTube	Video+Speech+Text
VAT-9	MAG-	Integrating Multimodal Information in Large Pretrained Transformers	ACL 2020	[coding]	CMU-MOSI, CMU-MOSEI	Video+Speech+Text
VAT-10	MTEE	Modality-Transferable Emotion Embeddings for Low-Resource Multimodal Emotion Recognition	AACL-IJCNLP 2020	[coding]	IEMOCAP, CMU-MOSEI	Visual + Text + Audion
VAT-11	Multimodal Routing	Multimodal Routing: Improving Local and Global Interpretability of Multimodal Language Analysis	EMNLP 2020	[coding]	IEMOCAP, CMU-MOSEI	Face+Speech+Text
VAT-12	SF-SSL	Jointly Fine-Tuning "BERT-like" Self Supervised Models to Improve Multimodal Speech Emotion Recognition	InterSpeech 2020	[coding]	IEMOCAP, CMU-MOSI, CMU-MOSEI	Video+Speech+Text
VAT-13		Multimodal Deep Learning Framework for Mental Disorder Recognition	FG 2020	[coding]	Bipolar Disorder Corpus (BDC)， Extended Distress Analysis Interview Corpus (E-DAIC)	Visual + Audio + Text
VAT-14	Deep-HOSeq	Deep Higher Order Sequence Fusion for Multimodal Sentiment Analysis	ICDM 2020	[coding]	CMU-MOSEI and CMU-MOSI	Visual + Audio + Text
VAT-15	DFF-ATMF	Complementary fusion of multi-features and multi-modalities in sentiment analysis	AAAI-WK 2020	[coding]	CMU-MOSEI and CMU-MOSI, IEMOCAP	Visual + Audio + Text
VAT-16	MSAF	MSAF: Multimodal Split Attention Fusion	2020	[coding]	RAVDESS, CMU-MOSEI, NTU RGB+D	Visual + Audio + Text
VAT-17	DCVDN	Visual-Texual Emotion Analysis With Deep Coupled Video and Danmu Neural Networks	2020 TMM		ISEAR, Multi-Domain Sentiment DataseT, [Video-Danmu]	Visual + Text
VAT-18	LMFN	Locally Confined Modality Fusion Network With a Global Perspective for Multimodal Human Affective Computing	2020 TMM		CMU-MOSEI, MELD, IEMOCAP	Visual + Audio + Language
VAT-19	SSE-FT	Multimodal Emotion Recognition With Transformer-Based Self Supervised Feature Fusion	2020 IEEE ACCESS	[coding]	CMU-MOSEI, IEMOCAP	Visual + Audio + Language
VAT-20	HPFN	Deep Multimodal Multilinear Fusion with High-order Polynomial Pooling	NeurIPS 2019	[coding]	CMU-MOSI, IEMOCAP	Visual+Audio+Text
VAT-21	MFM	Learning Factorized Multimodal Representations	ICLR 2019	[coding]	CMU-MOSI, IEMOCAP, POM, MOUD, ICT-MMMO, and YouTube	Visual+Audio+Text
VAT-22	MCTN	Found in translation: Learning robust joint representations by cyclic translations between modalities	AAAI 2019	[coding]	CMU-MOSI, ICTMMMO, and YouTube	Visual+Audio+Text
VAT-23	RAVEN	Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors	AAAI 2019	[coding]	CMU-MOSI,IEMOCAP	Visual+Audio+Text
VAT-24	HFFN	Divide, Conquer and Combine: Hierarchical Feature Fusion Network with Local and Global Perspectives for Multimodal Affective Computing	ACL 2019	[coding]	CMU-MOSEI, IEMOCAP	Visual + Audio + Language
VAT-25	DeepCU	DeepCU: Integrating both Common and Unique Latent Information for Multimodal Sentiment Analysis	IJCAI 2019	[coding]	CMU-MOSI, POM	Visual+Audio+Text
VAT-26	MMResLSTM	Mutual Correlation Attentive Factors in Dyadic Fusion Networks for Speech Emotion Recognition	MM 2019	[coding]	IEMOCAP, MELD	Audio+Text, Multimodal residual LSTM
VAT-27	CIA	Context-aware Interactive Attention for Multi-modal Sentiment and Emotion Analysis	EMNLP 2019		MOUD, MOSI, YouTube, ICT-MMMO, and MOSEI	Video+Audio+Text, Use a recurrent neural network to learn the inter-modal interaction among modalities with an auto-encoder.
VAT-28	CIM-MTL	Multi-task Learning for Multi-modal Emotion Recognition and Sentiment Analysis	NAACL 2019		MOSEI	Video+Audio+Text
VAT-29	LAMER	Learning Alignment for Multimodal Emotion Recognition from Speech	InterSpeech 2019	[coding]	IEMOCAP	Speech + Text
VAT-30	MHA	Speech emotion recognition using multi-hop attention mechanism	ICASSP 2019	[coding]	IEMOCAP	Audio + Text
VAT-31	MSER	Multimodal speech emotion recognition and ambiguity resolution	2019	[coding]	IEMOCAP	Audio + Text
VAT-32	EmoBed	EmoBed: Strengthening Monomodal Emotion Recognition via Training with Crossmodal Emotion Embeddings	2019 TAC		RECOLA, OMG-Emotion	Visual + Text
VAT-33	MulT	Multimodal Transformer for Unaligned Multimodal Language Sequences	2019 NIH Public Access	[coding]	CMU-MOSI & MOSEI, IEMOCAP	Visual + Text + Speech
VAT-34	MFN	Memory fusion network for multi-view sequential learning	AAAI 2018	[coding]	CMU-MOSI, ICT-MMMO,YouTube, MOUD, IEMOCAP and POM	Visual + Text + Speech
VAT-35	MARN	Multi-Attention Recurrent Network for Human Communication Comprehension	AAAI 2018	[coding]	CMU-MOSI, ICT-MMMO,YouTube, MOUD, IEMOCAP and POM	Visual + Text + Speech
VAT-36	Graph-MFN	Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph	ACL 2018	[coding]	[CMU-MOSEI]	Visual + Audio + Text
VAT-37	LMF	Efficient Low-rank Multimodal Fusion with Modality-Specific Factors	ACL 2018	[coding]	IEMOCAP, CMU-MOSEI, POM	Visual + Audio + Text, Perform multimodal fusion using low-rank tensors
VAT-38	RMFN	Multimodal Language Analysis with Recurrent Multistage Fusion	EMNLP 2018		IEMOCAP, CMU-MOSI, POM	Visual + Audio + Text
VAT-39	MRTN	Multimodal Relational Tensor Network for Sentiment and Emotion Classification	ACL-Challenge-HML 2018	[coding]	CMUMOSEI	Visual + Audio + Text
VAT-40		Convolutional attention networks for multimodal emotion recognition from speech and text data	ACL-Challenge-HML 2018		CMU-MOSEI	Speech + Text
VAT-41	HFusion	Multimodal sentiment analysis using hierarchical fusion with context modeling	2018 Knowledge-Based Systems	[coding]	MOSI, IEMOCAP	Visual + Audio + Text
VAT-42	BC-LSTM	Context-dependent sentiment analysis in user-generated videos	ACL 2017	[coding]	CMU-MOSI, MOUD, IEMOCAP	Audio+Video+Text
VAT-43	TFN	Tensor Fusion Networks for multimodal sentiment analysis	EMNLP 2017	[coding]	CMU-MOSI	Audio+Video+Text
VAT-44	MLMA	Multi-level multiple attentions for contextual multimodal sentiment analysis	ICDM 2017	[coding]	CMU-MOSI	Audio+Video+Text
VAT-45	MV-SLTM	Extending long short-term memory for multi-view structured learning	ECCV 2016		MMDB	Audio+Video+Text
VAT-46		Fusing audio, visual and textual clues for sentiment analysis from multimodal content	2016 Neurocomputing		YouTube dataset, SenticNet, EmoSenticNet	Visual + Audio + Text
VAT-47		Temporal Bayesian Fusion for Affect Sensing: Combining Video, Audio, and Lexical Modalities	2015 TC		SEMAINE	Face + Audio + Lexical features, Temporal Bayesian fusion
VAT-48		Towards an intelligent framework for multimodal affective data analysis	2015 Neural Networks		eNTERFACE	Visual + Audio + Text

Attribute-based

Index	Model	Paper	Year	Project	Dataset	Method
AB-1	MMDRBN	Knowledge-Augmented Multimodal Deep Regression Bayesian Networks for Emotion Video Tagging	2019 TMM		LIRIS-ACCEDE	Visual + Audio + Attribute
AB-2		Recognizing Induced Emotions of Movie Audiences From Multimodal Information	2019 TAC		LIRIS-ACCEDE	Visual + Audio + Dialogue + Attribute
AB-3		Multimodal emotional state recognition using sequence-dependent deep hierarchical features	2015 Neural Networks		FABO	Face + Upper-body
AB-4		Context-Sensitive Learning for Enhanced Audiovisual Emotion Classification	2012 TAC		IEMOCAP	Visual + Audio + Utterance
AB-5		Continuous Prediction of Spontaneous Affect from Multiple Cues and Modalities in Valence-Arousal Space	2011 TAC		SAL-DB	Face + Shoulder gesture + Audio

Aspect-based Network

Index	Model	Paper	Year	Project	Dataset	Method
ABN-1	MIMN	Multi-Interactive Memory Network for Aspect Based Multimodal Sentiment Analysis	AAAI 2019	[coding]	[Multi-ZOL]	Text+Aspect+Images, Aspect based multimodal sentiment analysis
ABN-2	VistaNet	VistaNet: Visual Aspect Attention Network for Multimodal Sentiment Analysis	AAAI 2019	[coding]	[Yelp-Food-Restaurants]	Visual+Text
ABN-3		Cooperative Multimodal Approach to Depression Detection in Twitter	AAAI 2019		Textual Depression Dataset, Multimodal Depression Dataset	Visual+Text, GRU+VGG-Net+COMMA
ABN-4	TomBERT	Adapting BERT for Target-Oriented Multimodal Sentiment Classification	IJCAI 2019	[coding]	Multimodal Twitter datasets	Image+Text, BERT-based
ABN-5		Predicting Emotions in User-Generated Videos	AAAI 2014		[Dataset]	Visual+Audio+Attribute, Video content recognition

Physiological Signal

Index	Model	Paper	Year	Project	Dataset	Method
PS-1		Emotion Recognition From Multimodal Physiological Signals Using a Regularized Deep Fusion of Kernel Machine	2020 TC			EEG + Other physiological signals
PS-2	MMResLSTM	Emotion Recognition using Multimodal Residual LSTM Network	MM 2019		DEAP	EEG + PPS(EOG+EMG), Multimodal residual LSTM
PS-3		EmotionMeter: A Multimodal Framework for Recognizing Human Emotions	2019 TC			EEG + Eye movements
PS-4		Personality-Aware Personalized Emotion Recognition from Physiological Signals	IJCAI 2018		ASCERTAIN	Personality+Physiological signals, Personalized emotion recognition
PS-5		Combining Facial Expression and Touch for Perceiving Emotional Valence	2018 TAC		KDEF	Face + Touch stimuli
PSB-6		Multi-modality weakly labeled sentiment learning based on Explicit Emotion Signal for Chinese microblog	2018 Neurocomputing			Face + Touch stimuli
PS-7		Analysis of EEG Signals and Facial Expressions for Continuous Emotion Detection	2016 TAC			Face + EEG signals
PS-8		Multi-modal emotion analysis from facial expressions and electroencephalogram	2016 Computer Vision and Image Understanding			Face + EEG
PS-9		Combining Eye Movements and EEG to Enhance Emotion Recognition	IJCAI 2015		[dataset]	Eye Movements+EEG signal
PS-10		Multimodal Emotion Recognition in Response to Videos	2012 TAC			Eye gaze + EEG signals
PS-11	HetEmotionNet	HetEmotionNet: Two-Stream Heterogeneous Graph Recurrent Neural Network for Multi-modal Emotion Recognition	MM 2021	[coding]	DEAP, MAHNOB-HCI	EEG+PPS+GTN, A novel two-stream heterogeneous graph recurrent neural network
PS-12		Simplifying Multimodal Emotion Recognition with Single Eye Movement Modality	MM 2021		SEED, SEED-IV, SEED-V	EEG+Eye movement signal, A generative adversarial network-based framework