Adaptive Multi-Dimensional and Multi-Scale Convolutional Fusion for Multimodal Sentiment Analysis*

The integration of multiple modalities significantly enhances the ability of sentiment analysis models to extract information. Unlike single modal approaches, multiple modalities can interact and complement each other, addressing the task from different perspectives. This highlights the critical role of multimodal fusion. However, most existing models are predominantly built on attention-based pairwise modality fusion techniques. Different combinations of these techniques tend to focus on different aspects, which often introduces noise and impedes the model’s ability to effectively identify critical information. To address this issue, we propose an Adaptive Multi-dimensional Multi-scale Convolutional Fusion network (AMMCF). Specifically, we treat different modalities as distinct information channels and use 3D convolutional neural networks to simultaneously integrate information from three modalities. Additionally, to enhance the model’s robustness, we incorporate multiple convolutional heads into the convolution operations, performing different convolutions across various dimensions. Finally, we obtain fused information from the modalities and utilize both the fused features and the unique features of each modality to accomplish the final prediction task through multiple convolutional layers. Extensive experiments on benchmark datasets demonstrate that this method achieves superior performance in multimodal sentiment analysis tasks.