Multimodal contrastive learning with neuroimaging and cognitive tests for Alzheimer’s disease diagnosis

Alzheimer’s disease (AD) is a neurological illness that causes cognitive impairment. Computer-aided diagnosis can help diagnose Alzheimer’s disease early before clinical symptoms appear. Currently, many deep learning methods show good performance in AD diagnosis. Still, most of these methods are based on single/multimodal neuroimaging, leading to a one-sided approach to disease modelling. Combining neuroimaging, cognitive tests, and demographics can significantly improve model performance and reduce the negative impact of noise. This study proposes a multimodal model that introduces contrastive learning, extracting and fusing feature representations separately from cognitive tests and neuroimaging data. After that, contrastive learning based on similarity is employed for both modalities’ features, assisting the network in learning cross-modal features. Moreover, the hybrid attention mechanism of the Transformer encoder is explored for feature fusion. Experimental results on 2082 cases from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset validate the effectiveness of our proposed multimodal model.