Emotion Prediction Using Multi-source Biosignals During Cognitive Behavior Therapy with Conversational Virtual Agents

Cognitive behavior therapy (CBT) is a mental health care approach that improves mood by spoken communication. In recent years, systems have been proposed to provide CBT using conversational virtual agents. Detecting emotions is expected to enhance the interaction between the participants and the system. Although an electroencephalogram (EEG) has been effectively used in emotion prediction, speech artifacts during CBT may affect performance. In this study, we use not only the EEG but also an electrocardiogram (ECG) and facial expressions to deal with the speech artifacts. To investigate the efficacy of fusing multi-source biosignals, we compared models using the EEG, and late fusion models that integrate the EEG, ECG, and facial expressions. Our results showed the late fusion models had higher concordance correlation coefficients than the models using EEG.