Contents Introduction
Recent advances in neuroscience and artificial intelligence prompted the development of brain-computer interface (BCI) [1]. BCI can record, decode and translate brain signals directly and build a pathway between brains and peripheral devices. Instead of evoking specific brain signals’ modalities relying on external visual, auditory, and sensory stimuli, motor BCIs can reflect people’s endogenous intentions to the voluntary movements, and thus provide a more natural and intuitive control.
Motor BCIs can be used for the restoration, compensation, and augmentation of motor function. Many patients suffer from diseases that cause motor disabilities, such as stroke, amyotrophic lateral sclerosis (ALS), spinal cord injury (SCI), Parkinson’s disease, and multiple sclerosis. For these patients, though they can only complete some in-balanced, distorted, or incomplete movements, intact or residual cerebral motor function areas can still send intentional commands. In this case, motor BCIs can decode movement intentions from brain signals directly, which can be further combined with orthosis, neuroprostheses, and robotic arm to be applied in neurorehabilitation and daily life assistance. Besides, active motor BCIs with neurofeedback can help patients rebuild the neuromuscular pathway and also facilitate neural plasticity [2]. For able person, they can also benefit from combing motor BCIs with exoskeletons for motor function augmentation or combining it with robotic arm to reduce physical burden. Although an able person keeps intact motor function, and thus detecting movement intentions from electromyography (EMG), sensors, or camera data is also feasible, motor BCIs have unique value because brain signals occur earlier.
BCIs can be invasive or noninvasive. Invasive BCIs detect brain signals from neuronal firing potentials or local field potentials, which are mainly studied in nonhuman primates and rodent animals, and also in clinical trials of humans as pilot studies [3]. Studies on invasive motor BCIs covered the motor behaviors of motor planning and executing, kinematics and kinetics parameters’ decoding, and neural encoding patterns in the motor cortex. For the practical application of motor BCIs, noninvasive recording from electroencephalography (EEG) signals has the advantages of convenience, low cost, less trauma, and not relying on medical and surgical expertise. Considering this reason, in this review, we concentrate on EEG-based motor BCIs.
According to the different ways of inducing motor function, the EEG-based motor BCIs can be categorized into motor imagery (MI) and motor execution (ME) [4]. MI corresponds to the mental rehearsal of motor action without any overt motor output [5]. During MI, the subjects are required to imagine the repetitive motion of certain body parts (usually hands, foot, or tongue), and each part can correspond to one specified output command. EEG oscillations of MI include sensorimotor rhythms (SMRs) or mu rhythms with the event-related power increase/ decrease known as event-related synchronization/ desynchronization (ERS/D). Different from MI, ME is more natural because the motion tasks of ME are consistent with the real movement intentions [6]. During ME, the subjects are required to move or attempt to move, such as opening hand, grasping, or moving hand in one direction. It is worth mentioning that it is difficult for most motor-impaired patients to execute desired movements. More often, they only keep some residual or completely-lost motor function. But even so, decoding movement intentions encoded in brain signals is still feasible. EEG oscillations of ME contain both the event-related potential and oscillatory components associated with the preparation and execution or attempted execution of voluntary motion [7].
To date, most of the studies on EEG-based motor BCIs involve upper-limb or lower-limb movements. For the upper limb, it is more proximal, which can imply better performance in brain signal decoding. Besides, compared with lower limbs, upper limb movement contains more kinds of movement types, and also the coordination of upper limbs, including hands, plays an important role in daily life. Controlling a wheelchair manually can also help patients with lower limb impairment. Thus, in this review, we mainly focus on the EEG-based motor BCIs for upper limb movement, including shoulder, arm, elbow, hand, wrist, and finger movement. Correspondingly, the removal of movement artifacts and electromyogram (EMG) artifacts mainly caused by proximal upper-limb movements were highlighted. Moreover, the decoding methods and future insights for the upper limb were introduced, which could be extended to the lower-limb movement as well.
To study a motor BCI, a typical flow is designing an experimental paradigm and collecting data, pre-processing signals, correlating the neural activity, extracting features and decoding movement intentions, and testing the decoding model online or further in real application systems. Though there are various reviews on the paradigms, or neural correlations, or algorithms, or applications of BCIs, no review has tried to give a comprehensive review on the motor BCIs of the upper limb. In this paper, following the flow, we aim to review the studies on EEG-based motor BCIs for upper limb movement as well as describe limitations and opportunities for developing more natural and practical motor BCIs. The organized flow of this review is presented in Fig. 1.
Experimental Paradigms
A. Experimental Paradigms of MI
The MI can be divided into kinesthetic MI and visual MI. The kinesthetic MI is supposed to involve kinesthetic experiences and to perceive muscle contractions and stretching mentally, whereas the visual MI requires the visualized imagination of action from a first-person or third-person perspective [8]. For MI-BCI, the most common movement cases are the imagined motion of separate body parts, including the left or right hand, foot, and tongue [9], [10], [11]. These movement cases are selected to generate enough discriminative neural activation patterns for classification. According to the somatotopic organization of the primary motor cortex (M1), movements of the same limb are usually with close activation regions of the cortex and share similar SMR modulations, which are hard to be discriminated from EEG signals. Thus, movements of different limbs are more beneficial to establish an effective MI-BCI. However, this setting can cause counterintuitive connections between movement actions and real output commands.
Recent studies turn more attention to establishing an intuitive MI-BCI. Edelman et al. [12] designed an experiment of complex motor imaginations consisting of right-hand flexion, extension, supination, and pronation movements. Similar upper-limb imagination paradigms include forearm extension, hand grasp, and wrist supination [13], [14], right-hand opening/closing [15], and finger movements [16]. Pereira et al. [17] investigated the EEG patterns of the interacting processes related to a single reach-and-grasp movement’s imagination task. Besides movement types’ decoding, some studies also focused on imagined body kinematics (IBK). In [18], Jeong et al. proposed an intuitive upper extremity imagery BCI, which decoded movement velocities of both imagined and executed arm-reaching tasks in six orthogonal directions. Kim et al. [19] decoded complex movement velocities during executed or observed/imagined movement tasks. Though these studies have explored to design a more intuitive and natural MI-BCI, some substantial works on improving the decoding performance, testing the systems online and in real applications, and also testing its with target-users, are still in need.
B. Experimental Paradigms of ME
Compared with MI, the ME is more natural because it decoded the real movement intentions related to actual movements. Studies on ME include movement type recognition, movement onset detection, and movement kinematics decoding. Movement types’ recognition is mainly studied for the sake of applications in daily life assistance, and thus its paradigms should be natural and easily transferable to people with residual or completely lost motor function. Common movements include elbow extension/flexion, forearm pronation/supination, palmar/lateral grasp, and hand opening/closing [20], [21], [22], [23]. Besides, in most daily scenarios, a grasp is combined with a reaching movement, and thus reach-and-grasp tasks are designed. The reach-and-grasp is a point-to-point movement, and when the hand reaches the target object, corresponding grasp actions are executed according to the types and shapes of objects, including movements of palmar grasp, pincer grasp, lateral grasp, wrist supination, and pinch [24], [25], [26], [27], [28]. Notably, when different objects are placed in different positions, this reach-and-grasp task-based BCI decoded neural information that encodes both grasp types and movement directions.
For movement onset detection, this kind of BCI discriminated intentions to move from the rest state [29], [30], [31]. Movement onset detection is vital for an online system to avoid false alarms. A two-staged strategy can be adopted by detecting movement onset first and then decoding movement types or kinematics [32] and [33]. Some studies also skipped the first stage by regarding the rest state as one class in movement decoding [28].
Both the movement types’ recognition and movement onset’s detection correspond to the movements’ discrete classification. For the movement kinematics decoding, it contained both the classification of direction, speed level, and torque-level and the continuous reconstruction of speed/velocity and position/trajectory parameters. Center-out is a classical paradigm for movement kinematics decoding. It is a point-to-point reaching movement, and the subjects are required to move the hand from the center point to one target point [34], [35], [36], [37]. Usually, the movement directions are orthogonal to each other in 2D or 3D space [34], [35]. This paradigm is basic and does not correspond to one specific application scenario, nevertheless, a basic paradigm is more beneficial to explore motor BCIs as a preliminary research and has better generalization performance. Numerous studies have adopted the center-out paradigm to study upper limb movement directions’ decoding [35], [38], [39], [40], [41]. Besides, Robinson et al. [42] performed a center-out paradigm at fast and slow speeds, decoded the speed level, and also reconstructed speed components in each axis. Though several studies also used the center-out paradigm for continuous kinematics reconstruction [18], [34], [41], [43], Wang et al. [44] discussed its limitations in continuous movement decoding and proposed an improved center-out paradigm, in which the subjects were asked to move the hands in four orthogonal directions at an angle of 45° from the cartesian coordinate axes.
For continuous kinematics decoding, pursuit tracking task (PTT) is widely explored [45], [46], [47]. In this experiment, the subjects are asked to control the cursor to track a continuously moving object. Kinematics parameters of positions, velocities, and accelerations are regressed. These studies indicated that continuous kinematic movement information was encoded in low-frequency EEG bands. However, several issues need to be noticed in continuous kinematics decoding. First, in the PTT experiment, eye movement along with the target cursor is inevitable and results in [45] suggested that the recorded position and velocity trajectories could be decoded during observed movements. Thus, decoding continuous movements from brain signals encoding motor function without visual information need further research. Second, the decoded movement profiles and low-frequency EEG signals are in the same frequency range, and thus the effectiveness of the linear model in interpreting its correlations is doubtable, which has been discussed in [19] and [48]. Finally, the continuous decoding performance in existing studies is still weak with the correlations between the decoded and recorded kinematics profiles being around 0.4. This is due to the low signal-to-noise ratio of EEG signals and also the difficulties in solving the regression problems. In this case, using EEG signals to decode the real-time position, velocity or acceleration of movement along with the sliding time windows is hard, which causes the unsatisfactory decoding performance of continuous movement. Thus, some solid work to demonstrate continuous movements can be decoded from EEG signals, and further improving its performance is necessary. Lately, in 2023, Wang et al. [49] proposed a Motion Trajectory Reconstruction Transformer (MTRT) model, and fused EEG signals with the geometric information of human joint points to improve continuous movement trajectory’s decoding, which provided an inspiring direction for continuous movement decoding.
Several representative experimental paradigms for EEG-based upper-limb motor BCIs, including MI BCIs and ME BCIs, are summarized in Fig. 2.
The summarization of experimental paradigms for upper-limb motor BCIs, including (a) Motor imagery (MI) BCI and (b) Motor execution (ME) BCI.
Neural Correlates
For a motor BCI, correlating neural activities with physical movement behavior is essential. This could not only help us know the brain activation patterns associated with movements but also provides instruction for developing the decoding model. Furthermore, by the neural correlates, we can inspect the collected and processed data for whether being contaminated by artifacts or not. Thus, before establishing the decoding model and further application, the plots of neural correlates are encouraged to be done.
A. Movement-Related Cortical Potentials (MRCPs)
It is known that the MRCPs encode movement-related information associated with the planning and execution or imagination of voluntary movement, and can be captured from low-frequency EEG signals (usually < 10 Hz). The MRCPs can be categorized as a visually-cue-based movement-related potential (MRP) or a self-paced Bereitschaftspotential (BP) associated with motor preparation [50]. Contingent negative variation (CNV) is also a cue-based potential, and CNV is time-locked to the cue onset and MRP is time-locked to the movement onset. To elicit the CNV, a preparation cue for subsequent movement instruction and an imperative cue to instruct the subjects to execute movements immediately are necessary. The illustration of CNV, MRP, and BP is presented in Fig. 3 (a).
(a) Illustration of contingent negative variation (CNV), movement-related potential (MRP), and Bereitschaftspotential (BP). (b) Different chronological components of BP, including the early BP, late BP, motor potential (MP), and movement-monitoring potential (MMP).
Both the cued MRP and self-paced BP contained three main chronological components, i.e., the early component, the late component, and the re-afferent potential [50], as depicted in Fig. 3 (b). The early component (early BP or MRP) occurs up to 2 s before movement onset and shows a slow negative deflection. It represents preparatory somatotopic organization mainly in the supplementary motor area (SMA) [51]. The late component consists of a late BP or MRP and a motor potential (MP). The late BP starts approximately 400~500 ms prior to movement onset with a sharp negative deflection. Until around 100 ms preceding the movement onset, the MP arises at this time and negatively peaks just after the movement onset. The late component occurs mainly in the contralateral primary motor cortex (M1) [52]. After the negative peaking, there is a positive rebound, which is known as re-afferent potential or movement-monitoring potential (MMP) and reflects neural modulations mainly in the primary somatosensory area (S1) [53]. The MMP could be related to the fine control of movement.
The readiness potential (RP, includes early-and-late BP or MRP) and MP correspond to the movement planning and execution, respectively. Detecting movement intentions from RP shows the feasibility to predict movements before the movement onset. The negative phase of MRCP is modulated by various factors such as the movement’s complexity and discreteness, exerted force or torque level, cerebellar lesion, and speed and precision of movement [54]. These differences imply the effectiveness of using the MRCPs to inspect movement. To visualize MRCPs, averaging technique is common because of the low signal-to-noise ratio of EEG signals and the relatively fixed time delay of elicit potential [55].
Previous studies showed that there were obvious MRCPs associated with unimanual and bimanual movements. In [24], [28], [56], and [57], larger negative offsets were observed in the centerline (Cz) compared to the contralateral or ipsilateral hemisphere. For unimanual movements, a contralateral effect was observed [39], [56], [57]. Studies in [28] and [39] showed significant differences between unimanual and bimanual movements. In [33], Jochumsen et al. showed that the MRCPs of palmar grasp tasks can be influenced by speed and force for both healthy subjects and stroke patients. In [24] and [56], MRCPs varied between different hand movement types. In [29], Bi et al. showed that the MRCPs varied between attended and distracted states during the upper-limb motion task.
B. Event-Related Oscillations
Besides the MRCPs, event-related desynchronization/ synchronization (ERD/ERS) is found in voluntary motion execution and mental motor imagery with frequency-specific neural modulations in M1 [12]. MRCPs and ERD/ERS reflect different aspects of sensorimotor modulations and provide complementary motor information. ERD and ERS are event-related and frequency band-specific power decrease and increase in the alpha/mu (8–13 Hz) and beta (13–30 Hz) rhythms over the sensorimotor or motor cortex [12]. These phenomena can be considered as a decrease or increase in the synchrony of neuronal populations [55].
For upper limb movements, ERD occurs with mu and central beta rhythms desynchronization during the motor preparation, execution, and imagery [5]. It starts attenuation before movement onset and over the contralateral hemisphere and becomes bilateral toward and during the hand movement [58].
Accompanied by ERD, ERS occurs as a power rebound after movement with a rest state [55]. ERS arises in the beta band over the ipsilateral hemisphere or near the midline. The maximum of the beta ERS coincides with the reduced excitability of motor cortex neurons [59]. Results in [60] demonstrated the ERD/ERS difference was related to action difference. Common ERD/ERS visualization methods include the band power method [61] and event-related spectral perturbation [62].
C. Source Imaging
While scalp EEG provides spatial neural information in sensor space, the measured scalp signals of electrodes do not directly indicate the distribution and location of sources. An active electrode on the scalp measures the local-part electric field generated by neurons spiking. Due to the volume conduction, the electric field of each source spreads. Thus, the electrode does not solely record the neuronal activities underlying it. Rather, each electrode could pick up electric signals from different sources [63]. Besides, the recorded scalp electrophysiological signals can be distorted when being conducted by the skull, scalp, and other tissues.
To localize the electric sources in the brain, we can estimate them by solving the forward and inverse problems. The EEG source localization methods have been reviewed in [64]. For the upper limb movement, EEG source imaging has been applied to observe the activation patterns in source space [37], [44], [45], [46], [65]. In [44], the M1, SMA, superior parietal lobule cortex, frontocentral cortex, and dorsomedial occipital cortex in bi-hemispheres were activated during hand movement execution in a center-out task. In [45], similar results were observed for the position and velocity information encoded during movement execution. In [46], parieto-occipital activation patterns were observed for the velocity and acceleration decoder of a hand PTT task.
Movement Decoding
A. Artifacts Removal
Recorded EEG signals are easily contaminated by environmental and physiological artifacts [66]. Thus, artifacts’ removal is essential for linking inherent neural activities with motor behavior. To deal with artifacts, three procedures including artifact avoidance, removal, and correction can be involved. First of all, unnecessary motion not associated with the experimental task should be avoided, and also the subjects are asked to fix their gaze on the screen to avoid unnecessary ocular movement. Though avoiding artifacts to obtain desired EEG signals is an ideal way, some physiological and non-physiological artifacts are still inevitable in practical operations. Hence, artifacts, including muscular artifacts, eye blink and ocular movement artifacts, cardiac artifacts, motion artifacts, electrodes pop and drift, and electromagnetic interference, should be removed for upper-limb BCIs.
1) Muscular Artifacts:
Muscular artifacts, also termed electromyogram (EMG) artifacts, are caused by contracting muscles. EMG artifacts can be detected from the whole scalp and exhibit less repetition pattern. The spectral band of muscular activity (~20–300 Hz) overlaps with beta and gamma rhythms, which encoded oscillated information of motion in EEG signals. To attenuate EMG artifacts, usual techniques include spatial filtering and independent component analysis (ICA).
2) Eye Movement Artifacts:
Eye movement artifacts correspond to the visual tracking of target stimuli. Both ocular movement and blink artifacts can lead to the EEG signals’ contamination, especially in the frontal area. One way to remove the eye movement artifacts is based on vertical and horizontal electrooculogram (EOG) signals. In [19], EEG signals are first projected into independent components using ICA, and then the components whose correlation coefficients with EOG channels are above the threshold (usually 0.4) are regarded as eye movement-related components and rejected. The cleaned components are finally projected back to obtain cleaned EEG signals. This method is linear and easy to be implemented, but it needs extra EOG electrodes. Similar methods can be found in [67] for the subspace subtraction algorithm and in [68] for the sparse generalized subspace subtraction algorithm. Other methods not requiring EOG electrodes include blind source separation methods, e.g. principal component analysis (PCA), canonical correlation analysis (CCA), and ICA [69]. Notably, in data analysis of motor BCIs, electrodes at the prefrontal area (e.g., AFz, AF3, and AF4) are usually excluded because of EOG contamination and less motor-related information.
3) Cardiac Artifacts:
Cardiac artifacts include pulse artifacts which can be introduced when electrodes are placed near pulsating vessels such as the scalp artery, and it is related to the pulsatile blood flow originating from the heart [70]. This pulse wave can be recognized with a frequency of around 1.2 Hz. Besides, electrocardiogram (ECG) measures the electrical activity of the heart and can be recorded along with cerebral activity. The ECG artifacts can be removed using a reference waveform, or by an automated interval histogram method without a separate ECG channel [71].
4) Motion Artifacts:
For motor BCIs, removing motion artifacts is of vital importance, but it is often overlooked in most studies. Different from muscle artifacts, which usually originate from the head and neck musculature during motion tasks, motion artifacts occur with limb or head movement. Motion artifacts can be characterized as low-frequency and large-amplitude oscillations. In contrast, muscle artifacts are observed in high frequencies. MRCPs are easy to be contaminated by motion artifacts in low frequency, and since the motion artifact occurs accompanied by limb movement and resembles the movement pace, it carries a lot of information about the movement. Hence, removing motion artifacts is important for decoding movement information from neural signals, instead of from motion artifacts. Castermans et al. [72] pointed out that motion artifacts could pollute EEG signals strongly and also mentioned that this step is absent in several published EEG analyses. Besides, work by Ló pez-Larraz et al. [73] suggested that artifacts could over-optimize the performance of motor BCI and result in a misleading link between the oscillatory activity with motion behavior. Even for stroke patients, motion artifacts still exist in the paretic arm’s movement with their residual motor ability [73]. However, techniques for removing motion artifacts using a specific treatment are still lacking. In [39], Wang et al. applied the artifact subspace reconstruction (ASR) algorithm proposed in [74] for motion artifact suppression and showed its effectiveness in removing irregular and large-magnitude artifacts. Some further work on motion artifact removal and revealing the correlation of motion artifacts with movement decoding are needed. Besides, more attention should be paid to treating motion artifact elimination as one vital part of data preprocessing, especially for upper-limb motor BCIs.
5) Non-Physiological Artifacts:
Besides physiological artifacts, non-physiological artifacts such as electromagnetic interference, pops, and drifts should be considered. The line noise of 50 Hz or 60 Hz can be suppressed by a notch filter or removed as one part of ICA. Pops and drift artifacts are caused by the impedance variations of scalp electrodes and can be attenuated using the high-variance electrode artifact removal algorithm [75].
6) Supplements:
Besides the above-mentioned methods to remove one particular kind of artifact, filters and wavelets are also widely used for filtering. Filters and wavelets can be applied to remove the noise at certain frequency bands which are not overlapped with the frequency band of interest, and simultaneously promote feature extraction in motor-related frequency bands. There are various popular filtering methods for EEG-based motor BCIs, for instance, fast Fourier transform (FFT), Butterworth filter, Chebyshev filter, and finite impulse response (FIR) filter. Compared with filters, wavelet transform shows a well time-frequency tradeoff. Not only removing artifacts, but artifact correction is also essential to help enrich useful data. For bad electrodes, linear interpolation with neighboring electrodes can be applied. It should be pointed out that there is no optimal method for artifact removal. Besides, while removing artifacts, some clean EEG data sacrifices. Even so, eliminating artifacts is still crucial to explore the neural correlates with movement behavior. EEG waveforms and neural signatures can be plotted as one visualization tool to inspect the effectiveness of artifact removal.
B. Feature Extraction and Decoding
1) Feature Extraction and Selection:
In motor-BCI systems, once raw EEG data are input, data processing is first applied to reduce the influence of artifacts and noise on EEG data, which would help increase the accuracy and robustness of BCIs. After data processing, feature extraction is conducted to capture the discriminated information according to the neural patterns of different movement tasks. And also, some studies in deep learning ignore this step by capturing complex characterization implicitly using an end-to-end model. The extracted features from EEG signals vary from time-domain features and frequency-domain features to time-frequency features, independent components, current sources, brain connectivity features, etc. Intrinsically, the extracted features are associated with the neural correlations of MRCPs, ERS/D oscillations, source analysis, and brain networks during motor tasks. Extracting features associated with neural activities makes the decoding model more interpretable.
For the extracted features, some researchers may involve feature (or channel) selection techniques to exclude redundant data. The feature selection can help the model learn useful information better and meanwhile prevent over-fitting and reduce the computational burden. Mutual information, analysis of variance (ANOVA), Fisher score, Pearson’s chi-square test, etc. can be used to filter features statistically. Instead of ranking the features using statistical characteristics, wrapper methods, e.g., forward or backward feature selection, and recursive feature elimination can select features by optimizing the object function of the decoding model. Besides, PCA and linear discrimination analysis (LDA) are also frequently used for the dimension reduction process.
2) Movement Decoding:
Based on the selected features, the movement intentions can be predicted (before movement onset) or estimated (after movement onset). The movement prediction makes sense in both the movement preparation period before Go-cue and also the movement reaction period after Go-cue and preceding movement onset. Even if movement estimation is conducted after movement onset, it is also of value for motor-injured patients with residual or completely-lost movement behavior.
From the perspective of the decoder, it can be divided into classifier and regressor, corresponding to the movement parameters’ classification and continuous regression, respectively. For the classification of motor BCIs, the popular classifiers in machine learning include the LDA, support vector machine (SVM), decision tree, k-nearest neighbor analysis (KNN), naïve Bayes classifier, etc. For the regression, it includes multivariate linear regression (MLR), partial least square (PLS), ridge regression, least absolute shrinkage and selection operator (LASSO) regression, Kalman filter, logistic regression, etc. In brief, the linear decoders could establish the simple linear link between the EEG data and movement parameters and improve the interpretability of models, and the nonlinear decodes could capture the nonlinear and non-stationary characteristics of EEG data. In addition to machine learning tools, neural network models in deep learning are widely explored for their remarkable learning ability. In the EEG-based motor BCIs, popular deep learning networks include the neural network (NN), convolutional neural network (CNN), long short-term memory network (LSTM), etc. Several models based on neural networks are well-known for their impressive performance in motor BCIs, e.g., the filter bank common spatial pattern (FBCSP) () [76], EEGNet [77], ShallowConvNet [78], DeepConvNet [78], etc. Besides, several trendy models are in vogue for their impressive performance, e.g., transformer [79], graph neural network [80], and deep belief network. It should be noted that most deep learning models are designed for MI-BCIs, and more attention should be paid to ME-BCIs considering their different neural patterns.
Apart from decoding movement intentions in a fixed window, developing a synchronized (cue-based) or asynchronized (self-paced) system for continuous movement detection using sliding windows is more meaningful. This procedure can reflect the decoding performance along with time, and results in [39] showed that the movement decoding performance kept steady during the movement preparation period and increased gradually with the decoding window involving more movement execution period. Furthermore, continuous pseudo-online or online decoding is more in line with the practical motor-BCIs systems.
Moreover, several studies are dedicated to dealing with the limitations of EEG data in developing a decoding model, e.g., small datasets, non-stationarity, and low signal-to-noise ratio. Brain signals are nonstationary and also vary greatly across subjects and sessions. Training a more robust decoder needs a larger amount of data. However, collecting EEG data can be time-consuming and tiring for subjects. Thus, several transferring techniques are introduced [81], e.g., developing cross-subject [82], [83] and cross-session [84] models. Generative adversarial network (GAN) [85] and variational auto-encoders (VAE) [13] are frequently used for data augmentation. Besides, the inherent non-stationarity of brain signals and electrodes’ drifts can cause the trained models in offline tests to have declining performance in online tests. Therefore, online adaptive learning models are proposed to cope with it [86].
Though numerous studies have explored improving the decoding algorithms for motor BCIs, testing the models in online or practical systems over both short periods and long periods attracts less attention, especially using deep learning models. The online system’s performance is determined by multiple criteria, including true positive rate, false positive rate, precision, response time, etc. To date, decoding models based on machine learning are still preferable for online systems. To advance deep learning in practical uses, a robust, less-computing, and precise model is waiting to be verified.
Application Systems
The application systems of upper-limb motor BCIs mainly include neurorehabilitation for motor recovery and assistive systems for motor substitution. The typical application systems reported in the literature are shown in Fig. 4.
A. Neurorehabilitation
Motor rehabilitation therapy aims to help restore or reorganize impaired motor function based on motor relearning and neuro-plasticity mechanisms. Traditional techniques like bilateral arm training and constraint-induced movement therapy show promising effects on improving upper-limb motor function. However, they require residual movement ability of the paretic limb. Robot-assisted rehabilitation systems, including orthoses, exoskeletons, and other robotic devices, have emerged to enable intensive and repetitive arm training by providing assistance and resistance forces. For robotic therapy, an active non-assist or active assist mode is advantageous over a passive mode by involving the central nervous system (CNS) functionality in an active way [87]. Besides physical therapy, functional electrical stimulation (EFS) is used as a rehabilitation way by inducing muscle contraction via electrical stimulation on the skin surface.
For the rehabilitation systems, subjects’ active participation in therapies is prone to promote proprioceptive feedback and improve rehabilitation [88], [89]. To assist the subjects actively, EEG-based BCIs have been developed as an effective tool to recognize motor state from CNS directly rather than from peripheral kinematic sensors, EMG, etc. [90]. Therefore, it is also appropriate for patients with severe motor function deficits. The combinations of BCIs with muscle stimulators or robotic devices can rebuild neuromuscular pathways and restore motor function by inducing neuroplasticity.
In this regard, pioneering work has been dedicated to designing neurorehabilitation systems using BCIs and evaluating their rehabilitation performance. In 1999, Lauer et al. [91] designed an implanted hand grasp neuroprosthesis controlled by EEG signals. Latterly, Pfurtscheller et al. [92] reported an EEG-controlled FES device for restoring hand grasp using noninvasive surface electrodes. In following-up studies, motor function improvement using the EEG-controlled closed-loop FES systems has been detected [93], [94], [95], [96]. Not only equipped with FES, the motor BCIs combined with robotic devices, e.g. orthoses, exoskeletons, and robotic arms, are also explored. In 2011, Ang et al. [97] designed an MI-BCI with online robotic feedback. Latterly, in 2014, an MI-BCI coupled with a Haptic Knob robot for arm rehabilitation was proposed [98]. Frolov et al. [99] designed an MI-controlled hand exoskeleton to open the affected hand for post-stroke rehabilitation. Cantillo-Negrete et al. [100] presented an MI-BCI coupled with a robotic orthosis for hand rehabilitation. Besides the BCIs combined with robotic devices or FES for neurorehabilitation, BCIs equipped with visual reality (VR) feedback have also been demonstrated and reported an improved motor function [101], [102], [103], [104].
The existing neurorehabilitation systems are mainly based on MI-BCIs, which have shown similar cortical activation patterns with actual movements [105]. Studies have shown that detecting executed movement intentions from residual or
attempted movements is also feasible, and it is more natural [20], [40]. Besides, in offline movement decoding, the decoding of motor execution showed better performance than MI [106]. Nevertheless, few studies have applied ME-BCIs to develop therapeutic rehabilitation systems and assess their rehabilitation performance with targeted users. Comani et al. [107] designed a neuro-motor rehabilitation system integrating EEG, virtual reality, and a passive robotic device, and involved three post-stroke patients executing rehabilitation tasks for the test. Though all patients showed increased participation in rehabilitation, the proposed robot-based rehabilitation system assisted passively. Antelis et al. [40] investigated the decoding of natural movement attempts of paralyzed upper limbs, and experimental results from six stroke patients showed that significant continuous decoding of the affected limb was obtained in 4 out of 6 patients. However, the rehabilitation performance assessment over time is still lacking. Some further work to design a natural neurorehabilitation system using ME-BCI to control actively should be explored in the future.
B. Assistive Systems
For patients suffering from muscular disorders or paralysis of the upper limb, the loss of motor functions can severely restrict their life independence and decrease life quality. Thus, besides motor rehabilitation, motor substitution driven by BCIs is valuable to provide patients with daily-life assistance. While meeting the needs of daily-life assistance for patients, using BCIs to control the FES-based prosthetic arm or robotic arm can generate the afferent-efferent feedback loop and further facilitate motor function restoration.
For upper-limb movement assistance, the reaching and grasping actions play a key role in activities of daily living. The reach-and-grasp movements’ decoding in offline tests has been demonstrated in both healthy subjects [24], [25], [57] and patients [20]. Moreover, Tavella et al. [108] presented a natural MI-BCI to control an FES neuroprosthesis and involved four healthy subjects to carry out daily grasping and handwriting tasks. Müller-Putz et al. [109] developed an EEG-controlled grasp neuroprostheses system and involved 15 participants with high spinal cord injury to control the neuroprostheses for grasping movements, including hand open, rotation, lateral and palmar grasp, and rest. This team also established an online natural simulation scenario based on an EEG-based ME-BCI and an avatar’s robotic arm, and 15 healthy subjects were recruited to control the avatar’s robotic arm and interact with virtual objects. Nann et al. [110] proposed a hybrid EEG-and-EOG-controlled whole-arm exoskeleton and evaluated its feasibility, safety, and user-friendliness by involving hemiplegic stroke patients in a drinking task. Besides, the feasibility of this hybrid BCI for the semi-autonomous whole-arm exoskeleton control was demonstrated with seven healthy subjects [111]. Recently, Tang et al. [112] developed a hybrid-controlled wearable robotic upper-limb system based on MI and object detection, and the online test verified the effectiveness of the system in assisting patients to complete target object grasping tasks. Besides, for paralyzed patients with motor disabilities in both upper and lower limbs, using an MI-BCI [113] or MI-based hybrid BCI [114], [115] to control the wheelchair can assist the mobility.
The applications of motor BCIs in daily life assistance could further help improve rehabilitation. Soekadar et al. [116] designed a brain-controlled hand exoskeleton to be applied in life scenarios, and results showed an improved rehabilitation performance of subjects in the ability to manipulate daily-life objects. Cheng et al. [117] investigated the ability of the MI-BCI-based soft robotic glove to incorporate with daily-living tasks for stroke rehabilitation, and randomized controlled experiments from eleven chronic stroke patients depicted the rehabilitation improvement.
Performing rehabilitation tasks in a natural scenario could help the participants engage in and activate the brain more excitedly. However, its comparison of brain activation patterns and rehabilitation performance with traditional therapeutic tasks has not been studied as far as we know. Combining daily-life assistance with neurorehabilitation in a natural living scenario is valuable to be explored in future work.
Future Insights and Discussion
In past decades, the development of EEG-based motor BCIs for upper-limb movements is flourishing, covering the aspects of motor-related brain activation patterns, movements’ decoding, and application systems controlled by motor BCIs. In Table I, we summarized several typical studies on EEG-based upper-limb motor BCIs, especially in recent 5 years. As the cognition of brain motor function and the decoding techniques mature, it is time to focus on how to develop a more natural and practical motor BCI. In this section, we highlight vital issues to develop natural and practical motor BCIs in the future, including developing target-users-oriented, distraction-robust, and multi-limbs motor BCIs and applying fusion techniques to promote the natural and practical motor BCIs.
A. Natural Motor BCIs
A natural motor BCI is closer to a practical application scenario and can reflect the participants’ intuitive and directly-corresponding movement intentions. Such a natural motor BCI is friendly, engaging, well-performance, and easy to use for the participants.
For motor BCIs, the ME-BCI is more natural than the MI-BCI. Instead of repetitive mental rehearsal, the ME-BCI decodes the actual or attempted movement intentions. Besides, studies reported better decoding performance of executed movements than that of imagery movements [106]. For the ME-BCIs, the basic experimental paradigms like the center-out movements can be conducted as preliminary research, while to develop a more natural and practical BCI, the experiment tasks related to the actual application scenarios are more engaging, e.g. reaching and grasping objects to drink or eat.
Apart from the natural experimental tasks, how to establish a natural BCI-controlled application system should be considered. First, the online systems should show well real-time movement decoding performance, such as high accuracy and fast response, and thus provide the participants with a positive training experience and facilitate proprioceptive sensory feedback. Besides, virtual reality can be combined with motor BCIs to display imagined or attempted movements, which can greatly augment the training enthusiasm of patients [118]. Moreover, rehabilitation and assistance devices should be designed to be flexible, convenient, and user-friendly [119]. In the future, more and more studies are welcome to develop natural and practical EEG-based motor BCIs.
B. The Gaps of Motor BCIs Between Healthy Subjects and Target Users
Though most studies in the field of EEG-based motor BCIs involved healthy subjects as a preliminary test, the target users of motor BCIs are mainly the disabled or patients with motor impairment. There are differences between the healthy subjects and target users for not only brain functions but also physical motor behavior. For users with brain injury, e.g., patients with traumatic brain injury, decoding movement intentions from signals recorded at the injured area is hard. However, few studies involving healthy subjects take this point into account. It means that the decoding models validated in the healthy subjects could be not appropriate for the real target users at all.
Besides, due to motor and cognition function disabilities, the movements executed by the patients are usually reacted-slowly, incomplete, distorted, or weak. The motor-related brain signals are correlated to the movement response time, speed, precision, torque level, etc. Therefore, there are differences in the motion’s encoding patterns between the healthy subjects and target users. Not only that, the factors like learning ability, physical power, fatigue, and brain disease should also be considered when developing neurorehabilitation or assistance systems for patients. In a nutshell, to promote the motor BCIs into practical applications, validating the proposed decoding models and systems with the real target uses is necessary.
C. Motor BCIs Robust to Distraction
When developing a motor BCI for motor-impaired individuals in real application systems, the users’ physiological states (e.g., fatigue) should be considered. Besides, the BCI systems are usually tested in laboratory environments, and when applying it to real application scenarios, the physical noise from the environment could distract the users. Apart from the mental and physical factors, when using the BCIs in daily life, the users could be in a multitask-oriented state, e.g., using a BCI and talking with others at the same time, or using a motor BCI for the paretic limb and simultaneously picking up a cup using the unaffected limb. In all, to develop a more natural and practical motor BCI, the distraction effects should be considered, rather than preventing the users from doing or thinking anything while using BCI.
We categorized the distraction effects on BCIs into four classes, i.e., the physiological, physical, motor, and cognition distraction. The physiological distraction corresponds to the users’ state, e.g., fatigue, stress, anxiety, etc. The physical distraction comes from the surrounding environment, e.g., illumination and environmental noise. The motor and cognition distractions are related to the multi-task operations of users. The motor distraction corresponds to the movement distraction from the other limbs, and the cognition distraction corresponds to the mental cognition process. Several studies have discussed the influence of distraction on neural activations and BCI performance [101], [120], [121], [122], [123].
One solution to cope with the distraction effects on the decoding model is to establish separate models in the attentive and distracted conditions and adopt a hierarchical strategy to first estimate whether there is a distraction and then call for the corresponding movement decoding models [29]. However, this strategy could be unsatisfactory if the attention state is hard to discriminate by EEG signals. Another solution is to develop a universal model, which can be robust to distraction [124].
D. Multi-Limbs Motor BCIs
Existing studies on upper-limb movement decoding are mainly limited to the single upper-limb movement and keep the opposite upper limb still. This setting is to avoid movement interference from the no-dominant upper limb and concentrate on exploring the relationships between neural correlates and dominant upper-limb movement behavior. However, in practice, bimanual (bilateral upper limbs) coordination is essential. For rehabilitation training, bilateral arm training can facilitate rehabilitation after stroke [125]. Bilateral movement training can facilitate cortical neural plasticity by reducing interhemispheric inhibition and increasing recruitments of the contra-lesional hemisphere and descending pathways. Compared with unilateral movements, the bilateral imagined or executed movements activated more areas of the brain, which therefore maximize the neuroplastic benefits [126]. For movement assistance, bimanual coordination is common in daily life, e.g., eating, twisting bottle caps, and pouring water into glasses. Furthermore, because of the variety of bilateral movements, decoding bilateral movements to control brain-machine systems can improve multidimensional control.
It should be noted that the brain activation patterns during bilateral movement are not the simple superimposition of unilateral movements. During the bilateral movements, the brain signals are more complex and with stronger nonlinear dynamics. Besides, the multi-class movement combinations also pose a challenge for bilateral movements’ decoding. In recent years, the bilateral movements’ decoding from EEG signals arose of attentions from several research teams. In 2018, Vuckovic et al. [127] discriminated unimanual and bimanual movements from EEG signals in both ME and MI tasks, and a significantly stronger ERD was observed for the bimanual ME compared to unimanual ME. In 2020, Schwarz et al. [28] discriminated the unimanual and bimanual reach-and-grasp actions. Besides decoding movement types, in 2021, Wang et al. [39] investigated the neural signatures and movement directions’ decoding of unimanual and bimanual movements. Afterward, this team also explored the feasibility to decode the dominant hand’s movement during bimanual movements [128]. Lately, to cope with the weak multi-class classification performance, this team proposed a neurophysiological signatures-driven deep learning model to discriminate the unimanual and bimanual movements [129]. In 2022, Jiang et al. [130] fused EEG and functional near-infrared spectroscopy (fNIRS) as bi-modal signals to characterize and discriminate the bimanual robot-assisted cyclical movements. Thereafter, this team classified the unimanual and bimanual coordination movements from EEG signals, including the leftward, middle-ward, and rightward tasks [131]. Based on this study, this group reconstructed the bimanual movement trajectories [132]. In 2023, this team reported a compound-limbs paradigm which integrated upper-limb swing movemets to improve the decoding of lower-limb stepping intentions [133]. To advance bimanual coordination to brain-machine interaction, Handelman et al. [134] explored completing a bimanual self-feeding task by manipulating and coordinating two Modular Prosthetic Limbs based on implanted microelectrode arrays and shared control. In future work, more studies on decoding bilateral movements online and validating them in real-time systems and with target users are requisite.
E. Fusion Techniques of Data, Model, and System
To develop a natural BCI using EEG signals, the decoding performance is vital. Nevertheless, the EEG signals are nonstationary and have a low signal-to-noise ratio, which severely limits their decoding performance. Though many techniques have strived to solve this problem, because of the inherent limitations of EEG data, it is not enough to improve the decoding performance only by optimizing the decoding algorithm. In this case, applying the fusion techniques can be an alternative solution.
From the perspective of data, it is feasible to fuse the EEG signals with prior information [135] or other signals [116]. For the fusion with the prior information, for example, different decoding models can be switched according to the participants’ states, such as being attentive or distracted, and this strategy could ease the influence of the human state’s changing on movement decoding. Besides, the EEG data can be fused with other physiological signals, e.g., EMG/EOG signals and eye tracking data, or non-physiological signals, e.g., computer vision or position sensor data, to enhance the decoding performance, if available. For example, for the participants with little residual motor ability, though the complete movement intentions can hardly be captured from EMG signals, utilizing EMG signals to detect the movement onset is feasible [40]. Then, EEG signals can be used for continuous movement intentions’ decoding, and this strategy can effectively reduce the false alarm rate of the decoding model.
From the perspective of the decoding model, the fusion of features can help the model learn the neural information associated with different aspects of brain activation patterns, e.g., fusing the MRCPs with ERS/D oscillations [129]. Besides, ensemble learning can enhance BCI performance by fusing different decoding algorithms [136].
Moreover, when developing the application systems, intelligent assistive systems or shared controllers can be applied to alleviate the users’ control burden by fusing human intelligence with machine intelligence. Introducing the shared controller, e.g., fuzzy controller, model predictive controller, and sliding mode controller, can improve the systems’ stability, robustness, and safety. For example, Wang et al. [137] proposed a shared control model to fuse machine intelligence and human intention automatically. Besides, a hybrid BCI based on multiple brain signals can be effective by making up for the shortcomings of one kind of BCI, such as the combinations of MI with error-related potentials (ErRP) and P300 as the control signals of a robotic arm [138], and MI training combined with ErRP to improve BCI performance [139].
In all, though the state-of-the-art performance of EEG-based motor BCIs is limited, this does not mean that the BCIs cannot be applied in practice. In fact, motor BCIs have unique values in translating movement intentions from brain signals directly. EEG recording way is also creditable for its high temporal resolution, safety, portability, low cost, etc. Thus, the EEG-based motor BCIs are of high value for real-time movement intentions’ decoding. One solution for promoting them into practical applications is the fusion techniques, including fusing EEG signals with other biomedical or non-physiological signals, fusing the BCIs with prior information, feature fusion, and ensemble learning, developing intelligent assistive systems and shared controllers in application systems, and using the hybrid BCIs.
Conclusion
In this review, we introduced state-of-the-art research on EEG-based upper-limb motor BCIs, including the experimental paradigms, neural correlates, movement decoding, and its application systems. Moreover, with advances in decoding techniques, research on EEG-based motor BCIs should no longer just focus on whether motor-related information can be decoded from EEG signals, but be committed to developing more natural and practical motor BCIs. To this end, we give several prospective insights on how to promote it, e.g., designing motor BCIs with natural paradigms and tasks, establishing and validating the systems with target users, developing multi-limbs BCI, considering distraction effects in practical application, and utilizing fusion techniques to improve the BCI or hybrid-BCI systems’ performance. In all, we expect more progress in advancing motor BCIs to a more natural and practical scenario in future studies.