Remote Assessment of Emotion Dynamics in Schizophrenia using a Multimodal Dialog System

Schizophrenia (SCZ) is a complex psychiatric disorder, often characterized by abnormal emotional regulation attributed to hypo-connectivity between the ventromedial prefrontal cortex and other cortical regions. However, emotion assessment in SCZ today largely remains in-clinic and reliant on clinicians’ subjective judgement. This contributes to long diagnostic timelines, increasing the burden on both patients and healthcare systems. Automated remote analysis of emotional expression could provide an objective and scalable approach to identifying key features of SCZ, such as blunted affect - a core negative symptom thereof. This paper describes a cloud-based multimodal dialog framework for remotely measuring and assessing emotion dynamics in SCZ to address this problem. 61 patients with SCZ and 72 healthy controls interacted with a virtual human guide as she engaged them in a conversational assessment that included describing a picture while simultaneously recording their audio and facial video. We extracted occurrence probabilities of seven emotions for each image frame and generated a vector time-series of facial expressions for each video, from which we computed a set of features describing emotional probability, variability, and temporal patterns. We find that emotion fluctuation, entropy and higher order moments of emotion occurrence probabilities are most effective at differentiating between SCZ patients and healthy controls with high effect sizes. These findings suggest that automated emotion analysis has the potential to enhance early detection and objective assessment of SCZ, providing valuable insights into the emotional deficits associated with the disorder.