mBGT: Encoding Brain Signals With Multimodal Brain Graph Transformer

Leveraging the multimodal brain signals collected from various electronic devices, such as electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) data, has been regarded as a promising technique for automated brain disease diagnosis. Existing studies on multimodal brain signal analysis mainly focus on data alignment and basic feature integration, yet fail to capture the spatiotemporal correlations amongst diverse signals. This limitation results in the learning of biased brain signal representations. To eliminate this limitation, we propose a novel multimodal Brain Graph Transformer model, named mBGT. By learning both temporal and spatial dynamics exhibited in multiple brain signals, mBGT derives high-quality multimodal representations of brain signals from heterogeneous data sources. Specifically, we construct a dynamic brain graph based on simultaneous EEG-fNIRS data, which effectively embeds the spatiotemporal correlations amongst brain signals. Thereafter, we propose a spatiotemporal information enhanced graph Transformer to encode the constructed brain graph, extensively exploring the functional interactions among different brain regions. Extensive experiments are conducted on a public dataset to demonstrate the advantages of mBGT in terms of the performance in downstream detection tasks, compared to multiple state-of-the-art baselines. These highlight the superiority of mBGT in learning effective representations for brain graphs from multimodal brain signals.