BER-SISF: Inter-segment Fusion Network Using Emotional Similarity Information for Bimodal Emotion Recognition

For bimodal emotion recognition (BER), a key challenge is to effectively aggregate information from different modalities to improve emotion recognition performance. In this study, we present a dual-modality emotion recognition model that utilizes a segment-level, inter-modal similarity-based fusion method alongside a joint emotion feature fusion technique. This approach leverages inter-modal similarities to create central representation vectors for input sequences, thereby boosting realtime emotion recognition robustness. The model also integrates modality-specific emotional information with cross-modal affective similarities for improved information aggregation. The proposed model is validated on the public benchmark dataset IEMOCAP and achieves weighted accuracies of 84.23% in fourclass emotion recognition tasks, respectively, surpassing other state-of-the-art methods.