Hierarchical Transformer With Auxiliary Learning for Subject-Independent Respiration Emotion Recognition

Respiration is modulated by human emotional activity. Emotion recognition using physiological signals has recently gained considerable attention. However, most existing studies primarily focus on using electroencephalogram (EEG) signals for emotion recognition. This article explores the potential of utilizing respiration signals collected by wearable devices for emotion recognition. We propose a hierarchical transformer model to effectively extract emotional information from respiration signals. Furthermore, we introduce gender classification as an auxiliary task to further improve the accuracy of emotion recognition. Specifically, a frame transformer is employed to capture emotional information across frames of respiration signals. The extracted frame-level features are subsequently fused with segment-level embeddings through a specially designed fusion layer. Next, separate segment transformers are employed for emotion and gender to extract segment-level information, with a cosine similarity loss applied to promote shared feature learning. Finally, distinct classifiers are used for emotion and gender classification. To evaluate the effectiveness of the proposed method, extensive experiments are conducted on the DEAP and MAHNOB-HCI datasets under a subject-independent setting. For the DEAP dataset, the average classification accuracies are 72.42% for valence and 73.91% for arousal, while for the MAHNOB-HCI dataset, they are 80.45% and 79.69%, respectively. Compared to other physiological signals, such as EEG, respiration signals exhibit comparable potential for emotion recognition.