A Multimodal Based Framework to Classify Parkinson Disease using Voice Signal and HandDraw

Parkinson's disease (PD) is a progressive neurological disorder characterized by late-stage symptoms that significantly impact patients' quality of life. Early diagnosis and treatment of PD are crucial for alleviating symptoms and slowing disease progression. However, accurately diagnosing PD is challenging due to the overlap of its symptoms with those of other disorders. This study proposes a multimodal framework that leverages voice signals and handwritten graphics for the diagnosis of PD. The approach involves preprocessing and segmentation of speech signals to spectrogram and the binarization of handwritten graphics (circle, spiral, and meander) for analysis, incorporating the signal-to-noise ratio (SNR). Three pre-trained deep convolutional neural networks (DCNNs) and an attention-based neural network model were trained on both datasets. The integration of speech signals and HandDraw images facilitated a comprehensive multimodal analysis. The proposed DCNN with an attention network achieved a classification accuracy of 98.75% on the multimodal dataset, outperforming other state-of-the-art approaches. These results demonstrate the potential of the framework for improving PD diagnosis.