MERNet: Multimodal Emotion Recognition in Conversation Network via Cross-Modal Knowledge Distillation and Contrastive Learning

Multimodal Emotion Recognition in Conversation (ERC) is a task of predicting the emotion of each utterance in a conversation by utilizing both verbal and non-verbal modalities. However, existing approaches often struggle to bridge cross-modal gaps, resulting in misaligned features and frequent misclassification of minority emotions into semantically similar majority emotions. To address these challenges, we propose MERNet, a framework that employs cross-modal knowledge distillation and contrastive learning to align multimodal features and effectively distinguish subtle emotions in conversations. Our framework consists of two stages: 1) guiding non-verbal modalities with the text modality to transfer knowledge and align their features, and 2) applying contrastive learning with emotion labels as anchors to distinguish subtle differences between similar emotions and address the class imbalance problem. Experiments conducted on two benchmark datasets, IEMOCAP and MELD, demonstrate that our MERNet outperforms existing state-of-the-art models.