An intelligent multi-modal affect recognition system for persistent and non-invasive personal health monitoring

This paper presents a multi-modal affect recognition system that is capable of effectively estimating human affective states through analyzing and fusing a number of non-invasive external cues. The proposed system consists of a probabilistic information fusion model based on the influence diagram and a set of data analysis, feature extraction and affect recognition modules for processing heterogeneous sensory data (e.g., visual appearance and speech) to accurately estimate human affective states. Experimental results demonstrate that the proposed affect recognition system can automatically and robustly recognize human affective states. Our results also prove that by fusing complementary information such as facial expression with voice intonation and/or body posture, the affect recognition performance can be boosted and outperform the performance based on each individual information. The proposed multi-modal affect recognition system can be integrated into existing remote health monitoring and rehabilitation systems to provide objective, non-intrusive, and persistent affect recognition, thus assisting patients' health monitoring and diagnosis in a natural living environment.