MDDMamba: A Model for Multi-Modal Depression Detection with a Memory-Saving Cross-Modal Attention Mechanism Based on Mamba

Depression is a severe mental health disorder that constitutes a significant global health challenge, necessitating efficient and accurate automated diagnostic tools. Traditional approaches to automated depression assessment using multi-modal data often face three significant difficulties: small sample sizes in available datasets, high computational cost, and difficulties in effectively integrating different modalities. In this paper, we first mitigate the problem of dataset quality by utilizing an estimated imputation strategy for missing values based on a known data augmentation method. Additionally, we introduce a multi-modal depression detection model named MDDMamba, designed to address the latter two challenges by integrating features from pre-trained emotion models with the memory-efficient selective state space model named Mamba. The MDDMamba model combines features from pre-trained emotion models for both text and audio, enhancing the extraction of emotional information. The core of the MDDMamba model is the CrossMamba module, which replaces the traditional Transformer’s quadratic computational complexity with a linear complexity alternative, facilitating efficient multi-modal fusion. Experimental results on CMDC and EATD-Corpus datasets show that the MDDMamba model achieves better classification and regression performance and stronger generalization capabilities in depression detection. Notably, the MDDMamba model achieves an F1 score of 0.94 and a Pearson correlation coefficient of 0.86 on the CMDC dataset while maintaining lower memory cost, outperforming most advanced multi-modal depression detection methods across various experimental scenarios.