Multimodal Emotion Recognition based on the Fusion of EEG Signals and Eye Movement Data

Emotion recognition research plays a crucial role in advancing brain-computer interfaces. However, many current methods either rely on a single modal signal or use a multimodal fusion approach that may not fully consider the balance of information between modalities. In this study, we propose a novel multimodal emotion recognition method that combines electroencephalograph (EEG) signals and eye movement data using a cosine loss fusion approach. To extract deep features from the two modalities, we utilize ResNet 18 with a gated attention mechanism for EEG feature extraction, and DenseNet for eye movement data feature extraction. The cosine loss function is employed to address inter-modal differences and ensure a balanced fusion of information. Our proposed method achieves an impressive classification accuracy of $91.16 \%$ for the emotions of happiness, fear, sadness, and neutrality on the SEED-IV dataset. This outperforms both EEG signals (74.48%) and eye movement data $(68.38 \%)$, demonstrating the effectiveness of our approach in emotion recognition tasks.