Motor Imagery Decoding Enhancement Based on Hybrid EEG-fNIRS Signals

This study explores the combination of electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) to enhance the decoding performance of motor imagery (MI) tasks for brain-computer interface (BCI). The experiment involved measuring 64 channels of EEG signals and 20 channels of fNIRS signals simultaneously during a task of the left-right hand MI. By combining these two types of signals, the study aimed to understand how feature fusion affected classification accuracy for MI. The EEG signals were filtered into three bands (<inline-formula> <tex-math notation="LaTeX">$\theta $ </tex-math></inline-formula>: 4–7 Hz, <inline-formula> <tex-math notation="LaTeX">$\alpha $ </tex-math></inline-formula>: 8–13 Hz, <inline-formula> <tex-math notation="LaTeX">$\beta $ </tex-math></inline-formula>: 14–30 Hz), while the fNIRS signals were filtered into 0.02-0.08 Hz to improve signal quality for subsequent analysis. The common spatial patterns (CSP) algorithm was utilized to extract features from both EEG and fNIRS signals. This allowed the researchers to create a fused signal with both EEG and fNIRS features that could then be processed using principal component analysis (PCA). Finally, the processed data was fed into a support vector machine (SVM) classifier, which improved the mean accuracy rate of MI to 92.25%. By comparing the classification accuracies obtained with fused and unfused segments of EEG and fNIRS signals, the study discovered that fusing the signals significantly improved classification accuracy by 5%-10%. Furthermore, analyzing the activated brain regions using fNIRS showed that the auxiliary motor cortex was significantly activated during MI. These results demonstrate that hybrid signals with a fusion strategy can enhance the stability and fault tolerance in BCI systems, making them valuable for practical applications.


I. INTRODUCTION
MI is a cognitive task that involves mentally imagining oneself performing a physical movement without actually moving. Combining MI with physical exercise has been shown to be beneficial for learning new skills and improving The associate editor coordinating the review of this manuscript and approving it for publication was Jason Gu . sports performance [1]. Additionally, neuroscience research has used MI as a paradigm to investigate brain activity prior to the execution of action [2], [3]. MI-based brain-computer interfaces (BCIs) have potential applications in restoring motor movements for individuals who are paralyzed, disabled, or have suffered a stroke [4], [5]. By utilizing MI to control external devices, such as robotic arms or computer cursors, individuals with motor disabilities can bypass their impaired neuromuscular system and interact with the world around them in new ways. This could greatly improve their quality of life and independence.
To create a BCI that is based on measuring MI, EEG signals are commonly used to record the activity of cortical neurons using scalp electrodes. This method has several advantages, including its non-invasiveness, portability, and ability for long-term monitoring. [6], [7], [8]. Additionally, EEG signals provide high temporal resolution, allowing for real-time measurement of MI which can be converted into control signals to assist with motor movements [6], [9], [10]. However, a single modal assessment of EEG induces some disadvantages such as the non-intuitive representation of brain activity and low spatial resolution. It is also extensively reported that better BCI performance can be achieved with multimodal analysis instead of standalone EEG signals [11], [12], [13], [14], [15]. On this basis, multimodal studies which assess both brain's electrical activity and hemodynamic activity attract much interest. Hemodynamic activity can be recorded by position emission tomography (PET), functional magnetic resonance imaging (fMRI), and fNIRS, whereas only fMRI and fNIRS can simultaneously do the recording with EEG devices. Despite the availability of multiple recording modalities, the combination of EEG-fNIRS is emerging as a promising approach due to its low cost, portability, less discomfort, low interference, and good spatio-temporal resolution [16], [17], [18].
However, simply measuring these signals isn't enough -we also need to extract useful information from them in order to train our machine learning algorithms to recognize specific patterns of brain activity. Different researchers have used different methods for doing this. For example, Fazli et al. recorded event-related desynchronizations (ERDs) and time average fNIRS concentration changes during MI whereas Ma et al. adopted EEG power spectral densities and fNIRS signals amplitudes [12], [19]. In addition, the peak amplitude of EEG and the mean values of oxygenated hemoglobins (HbO) and deoxygenated hemoglobins (Hb) for fNIRS can also be applied as the input to a linear discriminant analysis (LDA) classifier [15]. Based on the aforementioned fused features, although the MI classification accuracy or the movement recognition accuracy can range between 65%-95%, which is higher than the results derived from standalone EEG or fNIRS features, feature extraction should be specified into different segmentation in the time domain or frequency bands in the frequency domain. For standalone EEG-based BCI, the CSP algorithm is widely used for extracting spatial features by constructing spatial filters for differentiating various kinds of MI. However, searching for the best spatial filter still depends on the information in the temporal domain, which makes it to be sensitive to temporal noise [20]. Although a series of methods such as common spatial spectrum pattern (CSSP) or common sparse spectral spatial pattern (CSSSP) were proposed, a bandpass filter that confines the signal in a certain band can largely improve the effect of a CSP algorithm [21], [22]. In this way, filter bank common spatial pattern (FBCSP) with its variants was proposed, which can extract features in different bands simultaneously [20], [23], [24], [25], [26].
In this study, we collected fNIRS and EEG signals of MI of 15 healthy subjects at the same time. A new EEG fNIRS multi-mode MI decoding method is implemented: the CSP algorithm is used to extract two single-mode features respectively, and PCA reconstructs the fused features into a new set of features. Then support vector machine (SVM) is used as a classifier to compare the classification effect before and after feature fusion. In addition, we use NIRS statistical parametric mapping (SPM) to generate brain-activated images based on the MI contrast of left and right-hand. The rest of the paper is organized as follows: Section II explains the methods we used, including information about the subjects, data collection, experimental protocol, and data processing and analysis. In Section III, we present the experimental results, and in Section IV, we discuss these results. Finally, we provide a conclusion in Section V.

A. SUBJECTS
In this research, we used two datasets. The first dataset, which we collected ourselves, consisted of twenty healthy individuals from Wuyi University. There were 5 males and 15 females with an average age of 21.5 years old. All participants confirmed that they did not have any chronic physical or mental illnesses. Before the experiment began, we asked the participants to avoid consuming caffeine or alcohol for four hours prior to the recording and to refrain from any strenuous exercise. They also had to sign an informed consent form and declare that they had never participated in an EEG experiment before. The Institutional Review Committee of Jiangmen Central Hospital approved this study. The second dataset, Dataset B, is a collection of public EEG-fNIRS data sets [27]. It included 29 healthy subjects, 14 males and 15 females, with an average age of 28.5 years old. Twenty-eight of these subjects were right-handed, and one was left-handed.

B. EXPERIMENTAL PROTOCOL
In this experiment, we used a measuring cap that included a 64-channel EEG system (SynAmps2 from Neuroscan in El Paso, Texas, USA), as well as a 20-channel fNIRS acquisition system called OXYMON MK III from Artinis in Nijmegen, Netherlands ( Figure 1A). The EEG acquisition system is composed of 64 unipolar, 4 bipolar, and 2 high-level inputs, with a total of 70 leads, whose sampling rate of each lead is adapted to 1000 Hz in this experiment ( Figure 1B) whereas the fNIRS which emits two wavelengths of 762.0 nm and 845.5 nm with a sampling rate of 50 Hz. Two Intel (R) core (TM) i5-7200u duo 2.7 GHz hosts (Vostro 3710 from Dell, Texas, USA) are used to run fNIRS and EEG acquisition systems, respectively. Figure 1B and Figure 1C depict the arrangement of EEG electrodes and fNIRS sources and detectors. Prior to recording, we ensured that the impedance between the skin and 65278 VOLUME 11, 2023 Authorized licensed use limited to the terms of the applicable license agreement with IEEE. Restrictions apply. electrodes was below 20 k , and that the intensity of light reception for fNIRS was above 200. Participants were seated comfortably in a chair with their hands resting naturally on their thighs. They were then instructed to perform a hand motor imagery task based on random image stimuli, which involved alternating between a 16-second rest state and an 8-second task state. ( Figure 1D). The peak response of fNIRS signals appears about 8 seconds after the stimulation onset under the influence of hemodynamics [28]. Subjects were asked to imagine corresponding hand movements based on the images on the display. The image task was randomly provided with a total of 50 left-hand experiments and 50 righthand experiments. In terms of dataset B, the experimental procedure is similar to ours whereas the distribution of electrodes is different ( Figure 1C) and a 2-second visual guidance is added for each trial of MI ( Figure 1E). For more details, please refer to [27].

C. MULTIMODAL SIGNAL PROCESSING FOR MI CLASSIFICATION
To achieve high accuracy of MI classification, multimodal fused signals are processed following four steps: Step 1: signal preprocessing As MI feature representations are embedded by different frequency bands of EEG signals (θ: 4-7 Hz, α: 8-13 Hz, β: 14-30 Hz), sixth-order Butterworth band-pass filters worked as filter bank (FB) were carried out for filtering EEG signals into different bands A control group was also proposed in which the EEG signals were filtered into 4-30 Hz. Furthermore, fNIRS signals were also filtered by a sixth-order Butterworth band-pass filter. Currently, the filtering of fNIRS use different high pass filter or low pass [29], [30]. If bandpass filtering is chosen, the filtering range will be selected based on the required noise frequency reduction and different task frequencies [31], [32], [33], [34]. Considering that the task frequencies of our two datasets are 0.04 Hz (1/24) and 0.03 Hz (1/27-1/29), we have chosen 0.02 Hz-0.08 Hz bandpass filtering.
Step 2: CSP feature extraction algorithm The CSP algorithm [35] was used to extract features from multi-channel binary classification tasks. The principle of the CSP algorithm is to maximize the covariance difference between two types of data to achieve high discrimination features. In this experiment, EEG signals were collected from 64 channels whereas fNIRS signals were collected from 20 channels. Suppose a group of data with one hand is the matrix, where i is the label for a certain hand (i = 1 means lefthand, i = 2 means right-hand), j means the group number, and the total number of groups on the left-hand and right-hand is n (j ⩽ n). Then the covariance matrix corresponding to a single group of stimuli is calculated as follows [21]: Then the covariance matrix of the overall data is the sum of the average values of all single-group stimulus covariance matrices of the two data types. Assuming that A l is the average of the left-hand covariance and A r is the average of the right-hand covariance. after calculating the covariance matrix of overall data, it is necessary to decompose A all by eigenvalue, which is calculated as follows [36]: where matrix B is a diagonal matrix composed of A all eigenvalues and matrix P is the eigenvector corresponding to matrix B. The eigenvalues are then arranged into descending order, and a whitening matrix F is calculated as follows [35]: After obtaining the whitening matrix, the spatial filter will be constructed. The whitening matrix F is used to process the average value G l of the left-hand data covariance and the average value G r of the right-hand data covariance, respectively as follows [36]: By decomposing G l and G r , we can get: where Γ Is the eigenvector, α l and α r are the eigenvalues of G l and G r , respectively, and the summation of which is equal to the identity matrix E = α l + α r . Therefore, when the eigenvalue of G l reaches the maximum, its corresponding eigenvector Γ will minimize the eigenvalue α r of G r , when the eigenvalue of G l reaches the minimize, its corresponding eigenvector Γ will maximize the eigenvalue α r of G r . Thus, the rule can project the two types of signals to different regions as much as possible for the initial classification purpose. Finally, the corresponding projection matrix H can be represented as follows [35]: The filtered matrix 2m×t can be obtained from the original data D ch×t through the spatial filter H , which is calculated as follows [21]: At this time, the matrix 2m×t is the result of D ch×t filtering. Finally, 2m×t is calculated as follows [21]: where var j is the variance of row j in 2m×t . ξ j is the feature vector we want to extract. In this paper, we choose four pairs as the number of features [37].
Step 3: feature fusion After data preprocessing, signals in frequency band 4∼30 Hz or separated three frequency bands of 4-7 Hz, VOLUME 11, 2023  Step 4: PCA feature selection and SVM classifier In this paper, PCA recombines the original feature matrix ξ ij obtained from CSP into a new set of unrelated comprehensive feature matrices. Select a set of unit orthogonal bases P, so that after the original feature is transformed to this set of bases, the covariance between two fields is 0, while the variance of the field is as large as possible. The covariance matrix C can be calculated by the following formula [38]: where C is also a symmetric matrix. ξ ij gets the matrix Y after the base transformation of P, that is, Y = PX. Assuming that the covariance matrix of Y is D, the relationship between D and C is: Next, we diagonalize the covariance matrix C and arrange the diagonal elements from big to small, then the first K lines of P are the basis to be searched, where K is the dimension of the new irrelevant comprehensive eigenmatrix. According to the properties of a real symmetric matrix with n rows and n columns can be found n unit orthogonal eigenvectors, assuming that these n eigenvectors are e 1 , e 2 , . . . ,e n . Then a matrix L can be found L = (e 1 e 2 . . .e n ), orthogonal basis P = L T , new uncorrelated comprehensive characteristic matrix [38]: The new feature group is linearly uncorrelated, so in theory, it will produce good classification results. The principal component score is the cumulative contribution rate obtained through covariance. Our approach for generating a feature subset involves several steps. First, we identify the principal component with the highest score and retain it as a feature. Then, we add the principal component with the next highest score to our feature set and continue this process until we've included all of the principal components.
Once we have selected our subset of PCA features, we pass them to an SVM classifier [39] for classification. To ensure the accuracy of the classifier, we use tenfold cross-validation. We terminate the process when we find the feature subset that yields the highest classification accuracy. Data and methods used for MI classification comparison are shown in Table 1.

D. BRAIN ACTIVATION
The activation results of brain regions in this experiment were obtained using NIRS SPM (KAIST bioimaging signal processing laboratory in daejeon, south korea). It uses the modified bill lambert law (MBLL) based on the generalized linear model (GLM) [40]: optical density (od) refers to how much light is blocked or weakened as it passes through a substance. The amount of attenuation is affected by the absorption coefficient ε, the concentration of the substance (C), the differential path length factor (DPF), the linear distance between the fNIRS light source and detector (d), and an unknown factor (G) that accounts for the influence of brain tissue on near-infrared light absorption, scattering, and the shape of the light propagation path. To eliminate this unknown factor, we calculate the starting and ending concentrations of the substance and then subtract them from each other. This gives us the changes in both substance concentration and light attenuation, which we can express as od: the near-infrared light with HbO and Hb contents can be obtained at the incident wavelength λ 1 and reflection wavelength λ 2 , respectively [40]: where od λ 1 is the attenuation of light with an incident wavelength of λ 1 , od λ 2 is the attenuation of light with a reflection wavelength of λ 2 , and ε λ x HbX is the absorption coefficient of HbO or Hb at the corresponding wavelength. Further calculation can be obtained from [40]: where [HbX] is the change in HbO or Hb concentration. By setting a comparison matrix, the activation area of the left brain relative to the right brain can be obtained, while the activation area of the right brain equivalent to the left brain can be obtained.

III. EXPERIMENTAL RESULT
There is a clear relationship between the number of features selected in PCA and the corresponding principal component value ( Figure 2). Specifically, as the number of selected principal components increases, the amount of information retained from the original feature group also increases.
In terms of the statistics on the number of principal components selected by 20 subjects (Figure 3), we can observe that only a small percentage of subjects had principal components in the range of 1-10 or 31-40. This indicates that either too few or too many principal components may not be ideal for capturing the relevant information from the original feature group.
Interestingly, a significant proportion of subjects, 65%, had principal components in the range of 11-20. This suggests that this range of principal components may be optimal for retaining the relevant information while also reducing the dimensionality of the feature space.
Overall, these findings highlight the importance of carefully selecting the number of principal components in PCA to ensure that the relevant information is retained while avoiding overfitting or underfitting the data.
To verify the effectiveness of our PCA feature selection, we used SVM and KNN [42] classifiers to compare the classification accuracy before and after PCA (Figure 4). It can be seen that in the KNN classifier, the average classification accuracy after PCA (89.50 ± 7.1%) is higher than the average classification accuracy of simple fusion (61.00 ± 19.6%), and the average classification accuracy of FB-PCA (91.85 ± 4.2%) is higher than the average classification accuracy of PCA.
To illustrate the effectiveness of the hybrid feature to MI decoding, we compare the classification accuracy with separated EEG signals, fNIRS signals and the PCA ( Figure 5). Bandpass filters with 4-30 Hz and 0.02-0.08 Hz are exerted on EEG signals and fNIRS signals respectively and the fused feature is also processed with such two filters. It can be seen that after feature fusion, the mean classification accuracy of MI for all subjects with multimodal EEG-fNIRS (88.00 ± 6.16%) is significantly higher than that with independent EEG signals (76.00 ± 8.46%) or independent fNIRS signals (71.25 ± 8.56%).
To show the capability of FB and PCA in EEG data preprocessing for MI decoding, we compare the mean classification accuracy with three fusion methods ( Figure 5). To extract more subtle features within EEG signals, three bandpass filters (4-7 Hz, 8-13 Hz, 14-30 Hz) are applied respectively for EEG signals. It can be seen that the mean classification accuracy with the FB-PCA method for all subjects (92. 25 ± 4.99%) is significantly higher than that with the mere PCA method (88.00 ± 6.16%) and simple fusion method (55.25 ± 16.69%).
To verify the generalization of the preprocessing method, we use the same method to process both EEG and fNIRS signals in dataset B [27]. We compare the classification accuracy of MI decoding with separated EEG signals, fNIRS signals and a hybrid one deriving from dataset B ( Figure 5). It can be seen that after feature fusion, the mean classification accuracy of MI for all subjects with multimodal EEG-fNIRS (96.90 ± 1.59%) is significantly higher than that with separated EEG signals (92.24 ± 1.90%) or fNIRS signals (53.56 ± 10.59%). Furthermore, the classification accuracy can also be improved by adding FB and PCA in data preprocessing for dataset B which is also shown in Figure 5. To better compare the classification effects of the single mode and the three fusion methods, we calculated the F1-score of the five classification methods. The experimental results are shown in Table 2. It can be seen that the F1-score of FB-PCA (73.96%) is significantly higher than that of single mode and the other two fusion  methods, and PCA (73.24%) is also higher than the F1-score of simple fusion (32.63%).
To further support our classification results, we use fNIRS signals from dataset A ( Figure 6A) and dataset B ( Figure 6B) to study the activation of brain regions during MI. The active areas of MI with dataset A are channels 12, 17 and 18 (right premotor cortex and supplementary motor cortex, primary motor cortex) with MI of the left hand (left figure). In contrast, the right side figure shows that channels 2, 3 and 8 (left premotor and supplementary motor cortex, dorsolateral prefrontal cortex) are activated by right-hand MI. However, although the left-right hand MI is activated by the contralateral brain region, the subregion of left-right hand MI is different for the two datasets. For dataset B, the left side figure shows that channels 13, 16, 18 (right dorsolateral prefrontal cortex, primary motor cortex, premotor and supplementary motor cortex of the right brain) are activated by the left-hand MI, and the right side figure shows that channel 1, 2, 4, 5 (left dorsolateral prefrontal cortex,  the premotor and supplementary motor cortex of the left brain) are activated by the right hand MI. The results showed that there were significant differences in the activated brain regions when the subjects imagined the left-and right-hand movements.
The experimental results of this study are consistent with the existing research results on MI [43]. The auxiliary motor areas remain activated throughout the process of motor imagination, while the left and right brain areas are asymmetric, which may be related to the asymmetry of the brain [44].

IV. DISCUSSION
This study examined how the brain reacts when performing left-right hand motor imagery, using both EEG and fNIRS data to measure cortical activation. The researchers collected both types of signals at the same time, as simultaneous multimodal recording of brain activity has become increasingly popular in recent years, given that different recording methods reveal various mechanisms of brain function. [12], [14], [37]. For instance, EEG signals reflect assembled neural electrophysiology, so they have a high temporal resolution, as the firing rate of a neuron's burst state can exceed 200 Hz.  On the other hand, fMRI uses blood-oxygen-level-dependent imaging (BOLD) to detect hemodynamic processes, allowing for measurements of HbO and Hb differences based on differential magnetic susceptibility. This approach provides a relatively high spatial resolution but lower temporal resolution since it is sensitive only to the difference between two brain states. Similarly, fNIRS uses near-infrared light to estimate spectroscopic measurements of HbO and Hb variation at the cortical level. Combining EEG and fNIRS is ideal for multimodal BCIs since they are portable and require electrode caps whose density can be adjusted according to the number of leads. VOLUME 11, 2023 In this study, the mean classification accuracy of the left-right MI was improved with simultaneous EEG-fNIRS recordings (92.25 ± 4.99%) compared to standalone EEG or fNIRS signals (EEG: 76.00 ± 8.46%, fNIRS: 71.25 ± 8.56%). Such results are in accordance with others' findings that a 5% to 24.4% improvement of BCI classification accuracy can be achieved for fused EEG-fNIRS signals in MI tasks [12], [45], [46], [47]. The main reason for the classification accuracy improvement attributes to the complementary features of these two modalities considering the volume of information, the frequency domain of signals or the spatial distribution electrodes. In addition, EEG-fNIRS-based multimodal BCI systems for other tasks also outperform those single-modal BCI systems. However, Ge et al. proposed a similar EEG-fNIRS multimodal system with fewer leads for MI classification [45]. Three EEG channels and six pairs of fNIRS channels are selected for the MI task and the mean accuracy can reach 81.2%, outperforming that with a large number of electrodes [48], [49], [50], [51]. Such a comparison implies that the redundancy of signals may contribute to the reduction of classification accuracy. To avoid feature redundancy, we applied PCA to regroup the original features with specific correlations obtained from the CSP into a new group of unrelated features and the classification accuracy can be as high as 92.25 ± 4.99%.
In addition, Chiarelli et al. also proposed an EEG-fNIRS-based multimodal BCI system for MI tasks but with a deep neural network (DNN) classifier [13]. They found the performance of the DNN classifier is better than that of an SVM one. Hemoglobin change and the event-related synchronizations/event-related desynchronizations (ERSs/ERDs) values were directly fed to DNN classifier without any preprocessing. Such a result manifests the importance of feature redundancy during the preprocessing that augments the ability of the consequent classifier.
The hybrid EEG-fNIRS BCI can also improve the classification accuracy of brain states instead of a task in a certain state albeit a limited number of channels were applied. For example, Kwon et al. proposed a hybrid EEG-fNIRS BCI system for distinguishing brain states of mental arithmetic, right-hand MI and an idle state [52]. Compared with the left-right hand MI task, telling the discrimination of brain states seems rougher and the difference is more evident as the prefrontal and parietal cortex are responsible for arithmetic tasks whereas the cerebral dorsal prefrontal cortex is responsible for left-right MI task [53], [54].
In this study, the CSP is used to extract the features from both EEG signals and fNIRS signals according to the frequency bands of EEG and the concentrations of HbO and Hb, respectively. In this way, the proportion of the fused features is deterministic and the contribution of these features to the BCI is not clear although the comparison of classification accuracy with unimodal is completed. Hosin et al. proposed a notion of fusion level of EEG/fNIRS in the multimodal BCI for MI classification. They find that the fused features with a low proportion of EEG features achieve a high level of classification accuracy which is in contradiction with our results as the classification accuracy for mere EEG signals is better than that with unimodal features of fNIRS [55]. Such a contradiction may be related to the preprocessing methods. However, making sure the proportion of features in the multimodal BCI can not only benefit the BCI design, but also it can be advantageous to the exploring of neural mechanisms in MI.
Research has found that there is an internal connection between the brain and the left and right hands, and this connection exhibits asymmetry [56]. For those who are righthanded, it is generally the left hemisphere of the brain that is used for motor performance. Some studies suggested that the asymmetry of the motor cortex may be due to right-hand preference [56], [57]. Other studies show that the left-hand movements of subjects are more affected by the visual environment without considering handedness, which strongly suggests the complex relationship between brain asymmetry and handedness [58], [59]. In fact, the motor cortex is also activated during voluntary exercise and when observing other behavior [60], [61], [62], [63], [64], [65]. Brain asymmetry is also related to the proficiency of habitual hands, which is the result of the interaction between neural connections and molecular regulation [66]. Brain asymmetry is also common in other fields, such as hemispheric asymmetry, which is used to assess the efficacy of treating depression [67]. The power spectral density topographic map of EEG also has asymmetry [68]. The motor function of the human body interacts with vision, hearing, language and memory. Therefore, the asymmetry between left and right brain regions in our activation results is the result of the interaction between the subject's left-hand and right-hand proficiency and other cognitive processes.
To demonstrate the effectiveness of our approach, we compared our results with those of other studies conducted in the past three years (Table 3). Our findings showed that the classification accuracy achieved through multimodal analysis was higher than that achieved through analysis of EEG or fNIRS signals alone [69], [70], [71], [72], [73], [74]. This could be attributed to the fact that the best results obtained from the single-mode analysis are already included in the optimal results from the multimodal analysis. Jian et al. [75] used the independent decision path fusion (IDPF) method to improve the average classification accuracy to 70.32 ± 8.74% across four types of classification problems in their EEG-fNIRS joint study. Pac et al. [77] used the pearson correlation coefficient based feature selection (PCCFS) strategy to classify EEG features derived from both EEG and fNIRS, which increased the classification accuracy from 65.52% and 58.62% achieved individually to 79.31%. Kwak et al. [78] used fNIRS to guide attention networks (FGANet), and by combining the two signals, they improved the average accuracy to 78.59 ± 8.86%. Arshia et al. [79] conducted integrated vector phase analysis (VPA) based on EEG and fNIRS. The classification accuracy of support vector machine (SVM), convolutional neural network (CNN), deep neural network (DNN), and VPA (with dual threshold circles) reached 82%, 89%, 87%, and 86%, respectively. Moreover, Kaga et al. [76] used EEG to measure the latency and amplitude of NoGo-N2 and NoGo/Go-P3 tasks, and found that Oxy-HB in the right frontal cortex of children with ADHD was significantly reduced. Our brain activation study also yielded similar results, as significant changes in HbO and Hb signals were observed during motor imagery in both left-and right-brain regions. This supports the separability of our data and the reliability of our classification results.

V. CONCLUSION
In this paper, we propose a left-right hand MI decoding method based on multimodal signals. A combined preprocessing method considering CSP, FB and PCA provides high discrimination features for a SVM classifier so that the classification accuracy can reach more than 90% for both self-collected and public datasets. Such a result manifests that the combined preprocessing method can be a paradigm for multimodal BCI. In addition, we also used NIRS-SPM to show the activated areas during the MI experiment which is consistent with a large number of fMRI studies. From the perspective of activation regions, there are significant differences between left-hand and right-hand MI activation regions, which also supports our classification results from the basic neural basis of the human brain. It also shows that the premotor cortex and supplementary motor cortex, primary motor cortex play an important role in MI.

(Tao Xu and Zhengkang Zhou contributed equally to this work.)
YULIANG YANG was born in Jiangxi, China, in 1996. He received the bachelor's degree in electrical engineering from Wuyi University, Jiangmen, China, in 2018, where he is currently pursuing the master's degree in pattern recognition and intelligent systems. He is also a member of the Jiangmen Brain-Like Computation and Hybrid Intelligence Research Center. His current research interests include brain-like computing, pattern recognition, and hybrid intelligence. He is currently a Lecturer with the School of Computer Science and Electronic Engineering, University of Essex, Colchester, U.K. His research interests include computational neuroscience, brain-computer interface, machine learning, neurophysiological data analytics, and their practical applications. He is an Associate Editor of IEEE ACCESS and a Review Editor of Frontiers in Human Neuroscience. He served as a guest associate editor for several special issues related to his research interests. ANASTASIOS (TASSOS) BEZERIANOS (Senior Member, IEEE) received the Ph.D. degree in bioengineering from the University of Patras, Patras, Greece, in 1987, the degree in physics from Patras University, Patras, and the degree in telecommunication from Athens University, Athens, Greece. He is currently a Senior Researcher with the Centre for Research and Technology Hellas (CERTH), Hellenic Institute of Transport, Thessaloniki, Greece. He has been a Professor in medical physics with the Medical School, Patras University, since 2004; the Head of the Cognitive Engineering (COGEN) Laboratory, The N.1 Health Institute, National University of Singapore, Singapore, from 2012 to 2020; and a Visiting Professor with the Computer Science Department, The New South Wales University (NSWU), Canberra, ACT, Australia, from 2016 to 2020. His research interests include artificial intelligence and robotics to biomedical signal processing and brain imaging and mathematical biology and systems medicine and bioinformatics. His work is summarized in 165 journals and 240 conference proceedings publications, one book, and two patents. He has research collaborations with research institutes and universities in Japan, China, and Europe. He is a fellow of the European Alliance of Medical and Biological Engineering and Science (EAMBES) and the Founder and the Chairperson of the Biennial International Summer School on Emerging Technologies in Biomedicine and the International Workshop on Human Intelligence Machine Coexistence (HELIOS). He is also an Associate Editor of IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING (IEEE TNSRE), Neuroscience, and PLOS One. He is also a reviewer of several international scientific journals.
HONGTAO WANG (Senior Member, IEEE) received the Ph.D. degree in pattern recognition and intelligent systems from the South China University of Technology, Guangzhou, China. From 2017 to 2019, he was a Visiting Research Fellow with the National University of Singapore, Singapore. He is currently a Full Professor with the Faculty of Intelligent Manufacturing and the Deputy Director of the Discipline and Science & Technology Development Center, Wuyi University, Jiangmen, China, where he is also the Director of the Jiangmen Brain-Like Computation and Hybrid Intelligence Research Center. His current research interests include brain-like computation, pattern recognition, deep learning, and hybrid intelligence. He is an Editor of Brain-Apparatus Communication and has served a Guest Associate Editor for the Frontiers in Human Neuroscience.