Multimodal Emotion Recognition Based on Global Information Fusion in Conversations

Multimodal Emotion Recognition in Conversations (MERC) has garnered significant attention due to its potential applicability in various real-world scenarios. Key challenges in MERC include leveraging contextual information effectively, accurately identifying speakers, and fusing complementary features between modalities. Despite remarkable advancements by existing methods, two major challenges persist: 1) managing long-distance inter-modality dependencies, and 2) recognizing complementarities among heterogeneous modalities. To address these challenges, we propose a novel framework that combines a Multimodal Transformer (MulT) with an Extended Relational Graph Convolutional Network (RGCN). To demonstrate the superiority of our model, we conducted experiments on the widely used IEMOCAP multimodal emotion benchmark dataset. The experimental results reveal that our framework not only surpasses all baseline models but also shows relatively high generalization capabilities across various emotion classes.