Driver’s Speech Emotion Recognition for Smart Cockpit Based on a Self-Attention Deep Learning Framework

Speech emotion recognition (SER) has a great importance in the emotion-aware human-machine interaction of smart cockpit. In order to address the problem of poor performance of a single feature in speech emotion recognition task, a novel multi-head self-attentional based deep convolutional neural network with feature fusion (MSA-CNN-FF) is proposed in this paper. Fusion features consist of utterance-level acoustic features and frame-level features. Firstly, the frame-level features are encoded by CNN. Then, the time-frequency region associated with emotion is paid attention to by using a multi-head self-attention mechanism. Finally, the local learned features are combined with acoustic characteristics of speech to obtain the final feature representation. An experiment is performed on the IEMOCAP database, and results demonstrate the validity of MSA-CNN-FF that outperforms other state-of-the-art baseline networks.