TMMM: Transformer in Multimodal Sentiment Analysis under Missing Modalities

The cross-task gap presents a significant challenge for multimodal models because of the differences in input-output workflows. For instance, multimodal pre-trained transformers may encounter uni-modal data during testing. To mitigate the gap, this paper introduces a Transformer in Multimodal Sentiment Analysis under Missing Modalities (TMMM) aims to perform well using missing-modal data during testing. TMMM uses a missing multimodal training approach to prevent accuracy degradation in testing. At the same time, a new network architecture allows the model to reconstruct missing modalities during testing. Classification token fusion and Mixture-of-Experts structures further enhance the model’s performance. A pre-training method utilizing contrastive learning, which can construct negative samples with positive samples, is proposed to overcome insufficient labeled data. Our experiments demonstrated the effectiveness of TMMM on two datasets with no modalities missing, i.e., it consistently achieved the highest classification accuracy and Macro-F1, which outperformed the best state-of-the-art baseline on each dataset by about 2% and 2.5%. Additionally, TMMM usually performs better than other baselines on datasets with missing modalities during testing.