Multilevel Spatiotemporal Modeling Network for Emotion Recognition With Multimodal Physiological Signals

Emotion recognition leveraging multimodal physiological signals has emerged as a critical research focus in recent years. Nonetheless, the inherent spatial and temporal heterogeneity across modalities in multimodal emotion recognition frequently hampers effective modality representation and complicates cross-modal feature alignment, ultimately constraining performance advancements. To overcome these challenges, this article proposes the multilevel spatiotemporal modeling network (MSTM-Net), which tackles multimodal heterogeneity by employing multiscale spatial feature extraction alongside a hierarchical cross-modal fusion strategy. Specifically, we introduce the holistic spatial feature aggregator (HSFA) to extract both local (left and right hemispheres) and global spatial features from electroencephalography (EEG) signals, thereby capturing spatial characteristics crucial to emotional expression. Meanwhile, a hierarchical temporal fusion module (HTFM) is introduced to iteratively refine interactions across diverse modalities by employing a multimodal hierarchical fusion strategy. Moreover, a cross-modal bridging module is integrated to flexibly address the inherent heterogeneity and disparities between primary and auxiliary modalities. Comprehensive evaluations on the DEAP, MAHNOB-HCI, and MOCAS datasets reveal that MSTM-Net significantly outperforms state-of-the-art multimodal baseline models, substantiating its efficacy in mitigating the heterogeneity of multimodal physiological signals.