Transformer-Based Bimodal Emotion Recognition: Signal Measurement and Transformer Fusion Approach

Human emotion plays a fundamental role in cognition, decision-making, and social interactions. Electroencephalography (EEG) is a fundamental widely used neurophysiological measurement tool that records brain electrical activity, providing valuable insights into emotional states. This article presents a novel Transformer-based bimodal emotion recognition model that integrates EEG and facial expression data to enhance multimodal representation learning and emotion classification accuracy. A comprehensive evaluation is conducted by implementing and benchmarking various state-of-the-art unimodal and bimodal emotion recognition models as well as exploring multiple fusion techniques. The model’s performance is validated on two widely used datasets, DEAP and MAHNOB-HCI, where the proposed model outperforms existing approaches in both unimodal and bimodal configurations, achieving an approximate 4% higher accuracy for valence and arousal dimensions. Specifically, for the DEAP dataset, the model attains classification accuracies of 0.66 (valence) and 0.72 (arousal) on the training set and 0.65 (valence) and 0.66 (arousal) on the test set. For the MAHNOB-HCI dataset, it achieves 0.69 (valence) and 0.73 (arousal) on the training set and 0.61 (valence) and 0.68 (arousal) on the test set. Additionally, the proposed Transformer-based fusion method outperforms traditional feature-level and decision-level fusion techniques.