Transformer Based Multi-view Learning for Integrating Static and Dynamic Complementarity of Brain Function

Dynamic temporal information and static connectivity information derived from functional magnetic resonance imaging (fMRI) can assist in the diagnosis of neurological disorders. However, existing disease diagnosis methods primarily rely on information from a single view, neglecting the advantages of multi-view information fusion. In this work, we propose an end-to-end multi-view fusion method that pre-trains on one view of fMRI data and fine-tunes on another view for disease identification. First, the dynamic temporal information and static connectivity information are integrated during the pre-training stage based on the consistency between the two views, effectively combining complementary information from both data types to improve disease identification accuracy. Finally, in the fine-tuning stage, for different fine-tuning datasets, we combine the residual connections in the model with the self-attention mechanism through the hadamard product. This guides the learning process and can be seen as a form of regularization or inductive bias, enhancing the models ability to learn from the data. Experiments conducted on the ADHD-200 dataset demonstrate that: 1) our method effectively fuses temporal and connectivity information from fMRI, improving the accuracy of brain disorder identification; 2) analyzing the consistency between the two views validates the effectiveness of the pre-training strategy and its positive impact on accuracy; 3) the residual attention maps of the model fine-tuned with functional connectivity networks (FCN) capture distinct symmetrical connections, which align with the inherent symmetry of FCN, supporting the rationale for using the hadamard product.