MMRBN: Rule-Based Network for Multimodal Emotion Recognition

Human emotion is usually expressed in multiple modalities, like audio and text. Multimodal methods can boost Emotion Recognition. However, the relationship between audio and text, and their roles in emotion expression have not been fully studied, and hence hinder Multimodal Emotion Recognition (MER). In this work, taking into consideration of the above two things, we propose two rules for MER, which are Rule 1: The audio module should be more expressive than the text module, and Rule 2: The single-modality emotional representation should be dynamically fused into the multimodal emotion representation. Following these two rules, we design the corresponding rule-based multimodal network (MMRBN). Experiment result on the public dataset demonstrates the effectiveness of our proposed rules and MMRBN.