Simple Attention-Based Mechanism for Multimodal Emotion Classification

Recent advances in attention-based machine learning models have significantly enhanced performance in various classification tasks, including emotion recognition. In this work, we introduce two novel multimodal architectures that leverage attention-based techniques for emotion classification. We introduce a state-of-the-art attention-based multimodal architecture and a baseline architecture. Our state-of-the-art architecture utilises attention-based unimodal models to extract contextualised embeddings from each modality and an attention-based fusion technique. Performance metrics and rigorous error analysis indicate that unimodal systems trained solely on text or speech data underperform compared to our multimodal system, which integrates both modalities. Furthermore, our system significantly outperforms existing state-of-the-art multimodal systems using the same modalities by nearly 4%.