Classifier Design for EEG-fNIRS Bimodal Brain-Computer Interface Technology

Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) can measure the electrical and hemodynamic activity of the brain with a good spatial and temporal resolution, respectively. However, the limitations of bimodal BCI data analysis have become increasingly evident. Therefore, in this study we explore the fusion strategies of EEG-fNIRS signals based on the spatiotemporal characteristics of the signals and deep learning theories. The current paper employs a recurrence plot-based feature extraction approach and designs a signal mapping enhancement method based on spatial information. A classifier design method was investigated to promote the performance of the EEG-fNIRS bimodal BCI. We thus adopt a convolutional neural network-long short-term memory (CNN-LSTM) model with the integration of an attention mechanism to construct the final bimodal classifier. Experimental results demonstrate that the proposed bimodal classifier achieves higher accuracy compared to unimodal classifiers, providing new insights for classifier design in BCI technology.