Consistency, Uncertainty or Inconsistency Detection in Multimodal Emotion Recognition

Humans exploit several sensory channels to recognize emotions and combine the information coming from the different channels into a single perception. Emotion Perception (EP) is also closely related to the Theory of Mind (ToM), which includes processes that capture socially and emotionally related inputs; furthermore, it interprets their meaning and direct responses accordingly. In this paper, we present a first step towards recognizing incoherence in emotions that exploits a three-level cognitive architecture. Starting with multimodal emotion recognition, a decision-maker determines whether a situation of consistency, uncertainty, or inconsistency exists and ultimately attempts to identify which case occurs. The detection is based on a suitable vector representation, in the conceptual level of the architecture, of moods on Russel diagram. A system designed in this way can impact HRI in terms of effectiveness by allowing a robot to get an idea about the actual emotional state of the person it interacts with.