A Novel Multi-Window Spectrogram Augmentation Approach for Speech Emotion Recognition Using Deep Learning

Deep learning frameworks are more suitable than conventional machine learning frameworks for classifying emotions from speech signals. Two limitations of existing speech emotion recognition (SER) frameworks are overfitting on small training datasets and poor classification for real time prediction. In this work, a novel augmentation scheme based on multi-window augmentation of speech spectrograms and transfer learning by finetuning lower layers of pretrained networks is proposed to resolve these limitations. Multi-window augmentation of spectrograms is adopted as a strategy to increase the number speech spectrograms sets utilized for training. Then, pretrained networks are adapted for speech emotion recognition and finetuned with the generated training datasets to improve classification accuracy for real time emotion prediction. The novelty of this work is in utilizing multi-window spectrogram augmentation to increase the number of speech spectrogram samples for real-time speech emotion recognition. The Interactive Emotional Dyadic Motion Capture (IEMOCAP) database was utilized to generate two benchmark datasets; tagged the AHSN and AESN datasets. Findings showed that the proposed SER framework achieved 69.15% and 67.17% while predicting emotions from the AHSN and AESN datasets, respectively. Thus, improving the classification accuracy of the proposed deep learning method by 6% and 2% on the AHSN and AESN dataset, respectively, as compared to the baseline method. Therefore, our proposed augmentation approach for SER using deep learning can be useful in characterizing emotions of speakers during conversation.