Adaptive Temporal Attention with Key-frame Loss for Speech Emotion Recognition

Speech emotion recognition has received growing attention to improving the user-friendly experience, but it is still a difficult challenge. That is mainly due to the problem of uneven distribution of emotional information in speech samples. The overall state label of a sample is not sufficient to fully describe each part of the sample. In other words, each part contributes differently to the overall recognition. In this paper, we build unidirectional and bidirectional temporal attention networks to enhance the ability to extract essential features, and build the key-frame loss function to make the attention weight closer to 0 and 1. The purpose of using the attention module and key-frame loss is to extract the information that is effective for classification and filter out the unimportant information. A large number of experiments were conducted on the IEMOCAP dataset to verify the effectiveness of the model and loss function. The experimental results show that under the combined effect of the attention model and the loss function, our method has achieved better performance than the existing methods, with a weighted accuracy of 59.15% and an unweighted accuracy of 60.87%.