Multimodal Sentiment Analysis Based on Expert Mixing of Subtask Representations

Multimodal sentiment analysis has garnered significant attention for capturing and comprehending human emotions more authentically and comprehensively, multimodal sentiment analysis has garnered significant attention. However, existing multi-task methods under multimodal sentiment analysis tasks are primarily text-driven, fail to fully consider non-linguistic modality tasks and lack the ability to extract subtask information from fused representations. To maximize the utilization of non-linguistic modality information in multimodal data, this paper proposes a subtask representation selection module based on an expert mixture to enhance the feature representation capabilities of speech and image subtasks. The module consisted of two parts: an expert network and a gating network. The expert network contains multiple identical experts, each trained to process a specific feature subspace and extract specific subtask representations from the multimodal fused representation to enhance the subtask performance. The gating network consists of speech and image parts, which are responsible for learning the contributions of the corresponding experts to their respective tasks and performing weighted fusion to strengthen the emotional expressions of the speech and image modality subtasks. The experimental results on the CH-SIMS and CH-SIMSv2s datasets demonstrate that the proposed subtask module based on an expert mixture can effectively improve the performance of multimodal sentiment analysis.