Deep Learning Technique Using Hybrid Features for Speech Emotion Detection

Speech Emotion Recognition (SER) is the application where the speech data is considered for processing, and the emotion of the person speaking is detected and identified. It helps in early identification of anxiety and depression in the person, which are normally considered as angry or sad. However, the major challenge in this field is that most emotions overlap pitch and tone in the speech. To resolve this issue, this research proposes a novel approach to identify the emotion from speech by extracting the variety of features from speech data and then fusing them to form a single fused feature set, which includes the different characteristics to detect the emotions. The experiments are performed on the opensource dataset from Kaggle, and the dataset is augmented to enhance the quality of data by adding noise, stretching, shifting time, and pitch. The augmented data is used to extract features from the three different feature extraction techniques, which are used to extract the temporal, time, and energy features, and combine the features. This fused feature set is used to train the Deep Learning (DL) algorithms. Three DL techniques are used for classification, i.e., Convolutional Neural Networks (CNN), Deep Neural Networks (DNN), and a Hybrid model using CNN and Transformer. Among all the models, DNN outperforms by achieving the accuracy of $\mathbf{9 1. 8 4 \%}$ on test data and classifying 8 different emotions. DNN is also the fastest among all the models, hence DNN outperforms in time accuracy tradeoff. The utilization of the proposed approach in real-time can benefit various fields by identifying the emotions in speech with accurate results.