A New Evaluation Methodology for Speech Emotion Recognition With Confidence Output

Emotion recognition in speech signals has the objective to classify speaker emotions exactly as human listeners would do. However, common accuracy measures only consider the majority voting of the raters and therefore require a classification with a final hard decision. In the present paper, we thus propose a new accuracy measure for automatic speech emotion recognition, which takes into account the distribution of the labels of all raters for a speech sample or segment, and the respective distribution of the classifier's confidence output. Hence, by example of a well-known but newly labeled database we will discuss important annotation aspects regarding the design of a ground truth for emotional databases. Furthermore, we demonstrate experimental results of an example emotion recognition applying both state-of-the-art and the novel accuracy measurement approach.