MCAF: Multi-Headed Causal Attention Fusion Approach for Multimodal Sentiment Analysis

Multimodal sentiment analysis(MSA) aims to synthesize multiple pieces of information, including textual, visual, and auditory information, to more accurately infer sentiment polarity. Due to the existence of complex and diverse affective causal relationships between different modalities, they interact with each other and constitute a multidimensional expression of human emotions. In the face of unbalanced data, i.e., when there are significant differences in the number of samples between different categories, the attention mechanism of the model tends to focus on spurious correlations in the training data, seriously impairing the model’s generalization ability and leading to bias, unbalanced prediction, and false correlation identification problems. To address the above problems, this study proposes a new multimodal fusion method based on the "within-sample sampling" and "cross-sample sampling" front door adjustment methods in causal inference strategies: the multi-head causal attention fusion method (MCAF). This method aims to capture potential causal relationships between different modalities through different attention heads, helping the model focus more on key causal features and avoid paying too much attention to false correlations in the training data. This helps improve the generalization performance of the model, make the decision-making process of the model more interpretable, and enhance the accuracy and robustness of the model for multimodal sentiment analysis tasks, making it more suitable for practical application scenarios.