Attention-Driven Adaptive Deep Canonical Correlation Analysis for Multimodal Sentiment Analysis with EEG and Eye Movement Data

Multimodal sentiment analysis (MSA) has gained prominence for its ability to more comprehensively capture human emotions by leveraging different data sources. In this paper, we propose Adaptive Deep Canonical Correlation Analysis (Adaptive DCCA), a novel framework that integrates electroencephalography (EEG) and eye movement signals for improved sentiment classification. Unlike traditional DCCA, our approach incorporates an attention-based adaptive weighting mechanism, allowing each modality to dynamically contribute according to its relative importance. We evaluate the proposed model on two standard datasets, SEED-IV and SEED-V, under both subject-dependent and subject-independent conditions. Experimental results demonstrate that Adaptive DCCA substantially outperforms existing methods, achieving high accuracy and robustness across varying emotional categories. Notably, the adaptive weighting proves especially beneficial in exploiting the complementary information embedded in EEG and eye movement data, although eye movement signals remain more susceptible to cross-subject variability. The comprehensive confusion matrix analysis further corroborates the effectiveness of our approach in distinguishing fine-grained emotional states. Overall, this work highlights the untapped potential of combining physiological signals for emotion recognition and offers a flexible, high-performance framework that can be extended to other affective computing applications.