Multi-Modal Depression Monitoring System

Depression is one of the most common mental health illnesses that strongly requires a correct diagnosis. This paper suggests a new AI-based game-driven system for continuous monitoring and assessment of depression to reduce subjective bias. Our model employs a multi-modal approach that analyzes speech, text, and visual patterns to recognize depressive symptoms. Unlike traditional methods that focus on emotion recognition from a single modality, our approach integrates multiple modalities. The novel game uses deep learning models consisting of mainly CNNs to extract features from patient's facial video recorded during the gameplay, in a feature-level fusion strategy; then, these three types of features were fused by traditional hard majority voting, generating a 0.89 F1 score for the development set in the E-DAIC corpus. All these are significant improvements over the baseline models.