Multimodal Emotion Analysis Using Attention-Based Neural Networks

Emotions largely contribute to natural human-computer interaction, yet the nuanced nature of human expressions poses significant challenges. The proposed work presents a novel multimodal emotion analysis architecture. It is an attention-based neural network architecture that leverages audio, text, and visual features extracted as Wav2Vec 2.0 embeddings, All-MiniLM-L6-V2 based rich textual embeddings and ResNet-50 based visual embeddings. The evaluation of the proposed work is carried out on Multimodal Emotion Lines Dataset (MELD). The proposed architecture outperforms state-of-the-arts by significant margin by capturing contextual and temporal dependencies for individual modality followed by multi-head attention based efficient fusion. Its performance is measured in terms of accuracy, precision, recall and F1 score. Code is available at https://github.com/chhavidhiman1/Multimodal-Emotion-Analysis-