Research of Tri-modal Mandarin Emotion Recognition Based on Speech, Facial Expression and Body Gesture

In order to overcome the problem that the accuracy and efficiency of Mandarin emotion recognition is low based on singal-modal, a tri-modal Mandarin emotion recognition method based on the feature-level fusion of three important modal of speech, facial expression and body gesture is proposed. Firstly, different emotional features are extracted from speech, facial expressions and body gestures, and then the three modal emotion features are fused by IAGA method. Finally, SVM classifier is used to construct prediction model and emotion recognition is completed on it. The CHEAVD Chinease multi-modal emotion dataset is utilized to evaluate our proposed methods. Compared with the traditional mono-modal and dual-modal emotion recognition, the experimental results show that the dual-modal fusion has higher emotion recognition rate than the mono-modal, and the tri-modal fusion has higher emotion recognition rate than the dual-modal, which verifies the effectiveness of the tri-modal Mandarin emotion recognition