A Multi-Modal Approach to Sensing Human Emotion

We are witnessing a revolution in body area sensing, with applications ranging from biometric-based security to personalized healthcare to sports performance training, among numerous others. The key application driver has been the emergence of wireless and/or contact-free technology for sensing human physiology, motion, and posture. We posit an analogous revolution enabled by advancements in sensing of human emotion. The applications are similarly diverse, spanning social skills education, business intelligence, monitoring of doctor-patient dynamics, and numerous others. This paper explores the sensing foundations necessary to achieve reliable detection and classification of human emotions. The approach combines sensing of speech characteristics, natural language processing, facial landmark monitoring, and machine learning.