IIMCNet: Intra- and Inter-modality Correlation Network for Hybrid EEG-fNIRS Brain-Computer Interface

Hybrid Brain-Computer Interface (BCI) enhances accuracy and reliability by leveraging the complementary information provided by multi-modality signal fusion. EEG-fNIRS, a fusion of electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS), have emerged as the suitable techniques for real-world BCI applications due to their portability and economic viability. Existing methods typically focus on the high-level feature representation with late-fusion or early-fusion strategies during the recognition tasks. However, they usually overlook the joint feature extraction of both intra-modality and inter-modality, which is crucial for optimizing BCI performance. In this study, we introduce an Intra- and Inter-modality Correlation Network (IIMCNet) to integrate both the inherent features derived from individual modalities: EEG, deoxygenated hemoglobin (HbR), and oxygenated hemoglobin (HbO), as well as the cross-modality features between EEG-HbR, EEG-HbO, and HbR-HbO data. The intra-modality correlation features are generated using a late fusion method (Intra-net), which combines the uni-modality features extracted by E-Net and f-Net. Concurrently, the inter-modality correlation features are extracted using an early fusion method (Inter-net). Inter-net is consist of three dilated convolution-based C-Nets that focus on neurovascular coupling across modalities. Finally, three intra-modality features, three inter-modality features, and the concatenate hybrid feature are fed into deep supervision module to enhance robustness and accuracy. Experiment results demonstrate the IIMCNet exhibits superior performance compared to methods that rely solely on either intra-modality or inter-modality correlation networks. Furthermore, IIMCNet outperforms other state-of-the-art methods in motor imagery and mental arithmetic tasks, respectively. (The code is available at: github.com/Y-xiaoyang/IIMCNet).