A Multibranch Cross-Attention Transformer Model for Psychiatric Disorders Diagnosis Using Multi-Modal MRI Data

With powerful feature extraction and representation ability, deep learning (DL) methods have became effective approaches for brain disorders diagnosis based on magnetic resonance imaging (MRI) data. Due to the limitation of MRI scanners, it is difficult to obtain a MRI scan that contains both functional metabolic information and sufficient structural details. Therefore, it poses the challenges for DL methods to learn better spatio-temporal representations in single modality data for identifying brain disorders. Multimodal MRI data fusion, has become an effective way to merge the complementary information in multi modalities. However, most MRI fusion methods mix features extracted from the single modality data into fused representation by learning simple linear combination, which may not well merge the high resolution of structure into temporal function pattern in latent space. To solve this problem, we propose an efficient method to fuse the structural MRI (sMRI) and functional MRI (fMRI) data into joint representations to utilize the cross-modality complementary information. In proposed method, we introduce the ROIs convolution based on the brain atlas to extract the sparse structural features from sMRI data and align to the time-series in fMRI data. A Multibranch Cross-attention Transformer (MBCAT) model is proposed to fuse multi-modal ROI-aligned features and further extract spatiotemporal representation for brain disorders diagnosis.The experiments on identifying schizophrenia (SCZ) demonstrate that our proposed method achieves a significant improvement over the single modality data with only sMRI data and only fMRI data. Furthermore, the DL models trained by proposed method outperform the previous multimodal MRI fusion methods and achieve the average accuracy of 86.88%.