Speech Emotion Recognition Based on Deep Learning Network with Non-local Networks

Convolutional Neural Networks (CNN) based Speech Emotion Recognition (SER) has the problem of learning bias towards stronger emotions while giving less weight to relatively faint emotions. This lack of subtleness causes the misclassification of emotions that have similar arousal such as anger and joy. To solve this problem, a novel speaker-independent SER methods consist of CNN with non-local networks and attention-based Bi-directional Long Short-Term Memory (BiLSTM) is proposed. The non-local networks can capture the dependence between any two points globally to enhance the correlation of emotional features at the utterance level. The extracted trichannel Log-Mel spectrograms features include the raw of LogMel spectrums, the Log-Mel spectrums’ deltas and delta-deltas respectively. Finally long-term contextual dependencies, local information and non-local information which is obtained by nonlocal networks is contained in the extracted feature. It was tested on the Interactive Emotional Dyadic Motion Capture Database (IEMOCAP) dataset. The experimental results show that the WAR of SER is 66.85% and the UAR of SER is 61.01%. Compared with several newly proposed SER methods, the effectiveness of the proposed model is demonstrated.