Multimodal Emotion Reasoning Based on Multidimensional Orthogonal Fusion

Multimodal emotion recognition aims to recognize emotions through modal features such as acoustic, text, and visual modalities. Although existing multimodal emotion recognition algorithms have achieved good performance, in many tasks, features are extracted from the same video for acoustic, text, and visual modalities, which will lead to redundant information across modalities. To alleviate this issue, this paper proposes a multidimensional orthogonal fusion module, which orthogonalizes the input feature vectors to obtain pairwise orthogonal vectors, and concatenates the base vectors with the obtained orthogonal vectors to form fusion vectors. Experimental results demonstrate that the proposed algorithm outperforms mainstream multimodal emotion recognition algorithms. Ablation experiments show that the input sequence of modality affects the model's performance, with the model achieving optimal performance when the input sequence is visual features, speech features, text features, and personality features.