A Multimodal Fusion Depression Recognition Assisted Decision-Making System Based on EEG and Speech Signals

Depression, as a widely prevalent mental disorder, significantly impacts the psychological well-being of a substantial portion of the global population. However, existing diagnostic methods for depression possess certain limitations, rendering them potentially ineffective for large-scale depression detection or failing to adequately exploit the multimodal physiological and behavioral information of individuals with depressive disorders. To address these challenges, this study proposes an innovative approach involving the multimodal fusion of Electroencephalography (EEG) and speech signals, aiming to comprehensively leverage data for the decision-making process. From aspects such as time-frequency domain features, spatial features, and semantic features, this research extracts information and fuses it to construct a multi-modal feature space. Simultaneously, a feature-level fusion strategy is introduced and applied in practical scenarios, resulting in the creation of a multi-modal fusion depression recognition assisted decision-making system. This system assists clinical practitioners in making more comprehensive and reliable identifications and diagnoses of individuals with depressive disorders. Following rigorous testing of its functionality, the multi-modal recognition system exhibits feasibility and effectiveness in real-world medical environments, offering novel perspectives and efficacious methodologies for the detection of depressive disorders.