A Comprehensive Exploration of Fine-Tuning WavLM for Enhancing Speech Emotion Recognition

Speech Emotion Recognition (SER) is a pivotal area in Human-Computer Interaction (HCI) with numerous applications. Traditional SER models rely on supervised learning but face challenges due to limited labeled data and the subjective nature of emotions. Self-supervised learning (SSL) offers an alternative by leveraging unlabeled audio data. WavLM, a large-scale SSL audio model, has shown promise in various speech processing tasks. This paper investigates the fine-tuning of WavLM for SER, analyzing the impact of different segment of layers on performance. We conducted experiments on the IEMOCAP and RAVDESS datasets, comparing various fine-tuned WavLM models with Wav2Vec 2.0 and HuBERT as SSL baselines. Results reveal that WavLM outperforms SSL base-lines, with all-layer fine-tuned WavLM achieving state-of-the-art results. Interestingly, fine-tuning top layers significantly enhances performance, suggesting their role in encoding paralinguistic information. However, fine-tuning all layers remains superior. These findings shed light on optimizing SSL audio models for SER and highlight WavLM's potential in emotion recognition.