Deep Investigation of Intermediate Representations in Self-Supervised Learning Models for Speech Emotion Recognition

Self-supervised learning (SSL) for speech is effective for various downstream tasks. However, the strategies for selecting upstream models, layer depths where intermediate representations are passed from upstream to downstream, and downstream model structures should be clarified. In this study, SSL was applied to speech emotion recognition (SER), where paralinguistic information is relevant. We investigated the wav2vec 2.0, HuBERT, and WavLM models on two SER datasets: IEMOCAP and RAVDESS. Moreover, we compared different layer depths in the upstream model and examined mean and attention pooling in the downstream model. Our experimental results revealed the following: predictive models are preferred over contrastive models for paralinguistic tasks; paralinguistic information is incorporated in the middle layers; the information relevant to SER is not extremely localized in time; and a single layer includes sufficiently effective information compared with the weighted sum. Our best model outperformed previous models in the frozen upstream condition.