Speech Emotion Recognition via Swin-Transformer and Cross-Attention Fusion Model

Speech emotion recognition (SER) plays a pivotal role in affective computing and human-computer interaction, serving in scenarios such as intelligent voice assistants and mental health assessment. Despite its potential, current approaches often have difficulty extracting comprehensive semantic cues from speech. Single-modal feature extraction cannot fully exploit the complementary information between raw waveform data and spectral representations. Furthermore, mainstream fusion techniques, typically based on concatenation or fixed-weight averaging, limit the depth of cross-modal interaction. To overcome these issues, this study presents a SER architecture that integrates a fine-tuned Wav2Vec2 model with a spectrogram branch enhanced by cross-attention fusion. The design follows a dual-branch paradigm: one branch processes the waveform through Wav2Vec2 and a BiLSTM to capture temporal-semantic patterns; the other extracts multi-scale spectral features from log-Mel spectrograms via CNN layers and an improved Swin Transformer. A cross-attention module adaptively fuses both modalities, allowing richer feature interaction. Experiments on the IEMOCAP dataset validate that the proposed approach achieves competitive results, outperforming several advanced baselines.