Multimodal Emotion Recognition Exploiting Audio, Video and Textual Features with Crossmodal Attention

Multimodal emotion recognition (MER) aims to identify and understand human emotions by integrating data from diverse sources such as speech signals, facial expressions, and textual content. This process requires addressing the complexities of merging varied data modalities, each of which presents unique challenges, including noise, variability, and inconsistencies in how emotional cues are expressed and perceived. This paper proposes a unique approach to MER by utilizing three complementary modalities-audio, video, and text-to classify emotions into distinct categories: anger, fear, joy, and sadness. The paper presents pairwise crossmodal attention to enhance the integration of unimodal features, which are then combined and processed through an artificial n eural n etwork (ANN) for classification. A key contribution of this work is the creation of the MAViT (Multimodal Audio Video Text) dataset, comprising 1,002 multimodal samples. The experimental results highlight the superior performance of the multimodal approach, achieving an F1 score of 0.62. Unimodal results were also analyzed in detail, further demonstrating the advantages of integrating data from multiple modalities to provide a more accurate and nuanced understanding of emotions.