Multi-channel CNN-based emotion recognition using recurrence plot representations of speech

During the last decades, the problem of speech emotion recognition (SER) has gained the researchers’ interest. A variety of previous works aimed to address the SER problem by two main approaches: on the one hand there exist the feature-based combined with the machine learning classifiers methods, while on the other, novel pipelines employing deep learning classifiers have also proven to be effective for emotion recognition. However, the lack of the existing approaches in the exploitation of the dynamic nature of the voice and further, the speech, aroused our curiosity of how the mathematical tool of the recurrence quantification analysis (RQA) and its extracted recurrence plots (RPs) and features could treat the SER problem. Consequently, the purpose of this work is to exploit the RPs and through the use of the RQA features construct a multi-channel convolutional neural network (CNN) which can differentiate the primary emotions, described in the German EMODB database. Our experimental results proved that our proposed pipeline outperforms the existing literature’s classification results, reaching an unweighted average recall (UAR) score up to 94%.