Multimodal Fusion for Enhanced Parkinson's Disease Screening: Integrating Brain MRI, Hand Drawing, Facial Expressions, and Voice Analysis

Parkinson's disease (PD) profoundly impacts millions in Sri Lanka, emphasizing the importance of early detection for better patient outcomes. We introduce “NeuraTrace PD,” an innovative application for early PD screening, combining brain MRI, hand-drawn image analysis, facial expression assessment, and voice analysis. For brain MRI analysis, we employed a Graph Convolutional Network (GCN), achieving an impressive accuracy of 91% and identifying distinct graph connectivity patterns using graph theory. We also use a deep Convolutional Neural Network (CNN) to reach 90% accuracy in analyzing hand-drawn images, while optimizing image resolution for efficiency and information retention. Facial expression analysis is a vital component, leveraging CNN and Constrained Local Models (CLM) to accurately extract Facial Action Units (FAUs), which contribute to a remarkable 95% accuracy using the XGBoost machine learning model. We introduce a Machine Learning (ML) algorithm and advanced signal processing to enhance voice feature analysis, offering a thorough understanding and classification of PD patients. It reaches 99% accuracy. In summary, our approach holds significant promise for advancing early PD diagnosis, ultimately improving the quality of life for those affected by this challenging condition.