Enhancing Emotion Classification Through Speech and Correlated Emotional Sounds via a Variational Auto-Encoder Model with Prosodic Regularization

Recent studies have explored the development of empathetic systems, capable of engaging in human-like communication and support in daily life tasks. This envision primarily requires the correct classification of human emotions in speech. Nevertheless, it is challenged by the complexities of classifying emotionally similar or overlapped audio content characteristics, such as between “Neutral” and “Sad” emotions, or “Angry” and “Happy” emotions, representing more than 17.42 percent of misclassifications. Furthermore, existing emotional databases, while publicly available, often contain weakly emotional data, posing challenges for both human listeners and artificial classification systems. In this paper, we proposed an emotionally regularised variational auto-encoder model for the classification of emotional speech audios with correlated emotional sounds, particularly for emotions with overlapping audio attributes. The dataset proposed is a mixed dataset, created from a public well-known dataset, and extended with real added customised data. The customised data are distinctive emotional sounds, such as shouting, crying, giggling, etc. Our emotion classifier model can classify four emotions, such as “Angry”, “Sad”, “Happy” and “Normal”. The sounds selected are associated with the selected emotions and the speaker gender. Throughout the training of the variational auto-encoder model and the classifier, we aligned our complex data with prosodic features, to obtain a more emotionally expected quality. The results show that the emotional sounds in speech and the speech utterances, are strongly complementary for a real emotional speech classification with a test accuracy of 0.973, improving the vanilla VAE with 0.881, and the standard CNN-based model with 0.652.