Influences of languages in speech emotion recognition: A comparative study using Malay, English and Mandarin languages

Emotion recognition plays a significant role in affective computing and adds value to machine intelligence. While the emotional state of a person can be manifested in different ways such as facial expressions, gestures, movements and postures, recognition of emotion from speech has gathered much interest over others. However, after years of research, recognizing the emotional state of individuals from their speech as accurately as possible still remains a challenging task. This motivates an attempt to study to understand the factors that influence identification of Speech Emotion Recognition (SER) such as gender and age. The aim of this study is to investigate whether a SER system can identify the emotional state of a person regardless of the language used. To investigate the influence of languages in SER, we explored how spoken expressions of four selected emotions (anger, sad, happiness and neutral) varied in the three languages of interest; Malay, English and Mandarin. In addition, the perceptual outcomes were studied in relation to identifying the advantage of speech emotion expression produced by native speakers. While supporting the fact that SER is language independent, the study reveals that there are language specific differences in emotion recognition in which English shows a higher recognition rate compared to Malay and Mandarin. This study also demonstrated that emotions expressed by native speakers have higher accuracy rates.