Enhancing End-to-End Speech Emotion Recognition Using Dual-Stream with Multi-Task Learning

Since speech emotion is variable and continuous, and human-computer interaction scenarios such as voice assistants are increasingly appearing in life with the development of technology, it is both necessary and challenging to allow machines to accurately recognise emotion in speech. In this paper, we propose a new end-to-end speech emotion recognition (SER) model, using dual-stream and multitask learning. The dual-stream model is constructed based on the upper stream, used for learning spec features, and the lower stream, built through a pretrained Wav2Vec 2.0 block. In addition, We introduce two tasks, gender classification and the Automatic Speech Recognition (ASR), to form a multitask learning mechanism together with the main task of SER. Specifically, we pass the input audio to a pretrained model and a feature extractor so that the pretrained model learns potential hidden features in speech directly, while the feature extractor extracts speech as specific features and passes them to the subsequent model. On the IEMOCAP dataset, our model has UA of 79.6% and WA of 82.6%. Comparing with the latest end-to-end model, our model improves 2.5% on UA and 2.3% on WA. Compared with other models, our model outperforms existing methods by 0.5% UA and 1% WA.