Detecting Distress Changes Using Multimodal Data During Interaction with A Smart Speaker

Mental health has a huge impact on humans, affecting both psychological and physical well-being. Excessive stress can lead to depression, reduced productivity, and even suicidal tendencies. Stress also impacts appetite and sleep quality, potentially leading to other health issues. However, stress accumulation often goes unnoticed until it severely impacts health, highlighting the need for daily stress level assessment. This study aims to estimate daily distress levels through natural conversations with a smart speaker. We utilize the audio-visual data of users interacting with a smart speaker on a daily basis, extract features from different modalities through analysis, and predict distress changes in daily life using questionnaire responses as labels. In the experiment, participants interacted with a smart speaker placed in their bedrooms, simulating daily life. Webcam recordings captured facial expressions, voice, and heart rate data, which were preprocessed for analysis. Predictions for happiness, depression, and anxiety levels were made using data from questionnaires filled out after each recording session, with scores ranging from 0 to 18. Results from the 14-day experiment with seven participants, aged 22 to 24, revealed MAEs of 2.04, 2.59, and 2.31 for happiness, depression, and anxiety levels, respectively. The corresponding RMSEs were 2.63, 3.20, and 2.91.