mDRA: A Multimodal Depression Risk Assessment Model Using Audio and Text

This letter proposes a multimodal depression risk assessment (mDRA) framework to overcome the limitations of single-modal approaches and data fusion in depression detection from audio and text. The mDRA leverages advanced neural network models, including a BiLSTM model with ELMO embeddings for text analysis and a ResNet50 model to extract 2D heatmap features from audio inputs. A collaborative attention mechanism integrates cross-modal information to enhance diagnostic precision. Experiments on the EATD-Corpus (Chinese language) and DAIC-WoZ (English language) datasets demonstrate the framework's effectiveness, achieving F1-scores of 0.94 (18% improvement) and 0.88 (3% improvement), outperforming state-of-the-art methods. These improvements enhance the reliability of early depression risk screening across different languages, which is critical for timely intervention. When deployed on a cloud platform with a mobile interface, mDRA can facilitate user-friendly depression screening through interactive virtual conversations, showcasing potential for early-stage assessment.