Enhancing Emotion Recognition using Contrastive Learning Representation of Paired Peripheral Physiological and EEG Signal

Human emotions play a fundamental role in human-computer interaction, and accurately recognizing emotions is essential for enhancing user-friendly experiences in HCI applications. Emotions are complex and cannot be simply identified by a single modality, making multimodal approaches crucial. In this study, we propose a cross-modal representation learning approach for emotion recognition using peripheral physiological signals (PPS) and electroencephalogram (EEG). We focus on leveraging PPS, which is easier to obtain in real-life scenarios due to wearable technologies’ availability. Through contrastive learning with paired EEG signals, we aim to enhance the capabilities of PPS representation and achieve comparable performance to EEG-based models in downstream unsegmented emotion recognition tasks. We utilized the DEAP dataset and low or high arousal category for the downstream task. The results of the study demonstrate the successful learning of PPS representations that closely resemble EEG signals representations, leading to comparable performance on downstream tasks.