Belief Mismatch Coefficient (BMC): A Novel Interpretable Measure of Prediction Accuracy for Ambiguous Emotion States

Despite of efforts made to model emotion ambiguity and develop ambiguity aware emotion prediction systems, there is a need for a quantitative and interpretable measure of the accuracy of such systems, regardless of recent advances in representing emotion ambiguity through probability distributions. In this paper, we propose a novel measure called the “Belief Mismatch Coefficient (BMC) that quantifies the differences in the belief that emotional states are perceived from certain regions within the arousal/valence space when comparing a predicted distribution to an underlying distribution inferred from ground truth ratings. The proposed metric is validated using simulated labels to demonstrate its effectiveness in quantifying various prediction errors. Furthermore, it is extended to real-case emotion prediction systems using two state-of-the-art modeling techniques on the RECOLA dataset. The experimental results confirm that the proposed metric can efficiently capture and differentiate between various prediction errors, while also offering insights into the predictions. Moreover, it demonstrates significant advantages in capturing a comprehensive view of the predicted distribution compared to traditional metrics such as Concordance Correlation Coefficients.