Real-Time EEG–EMG Human–Machine Interface-Based Control System for a Lower-Limb Exoskeleton

This article presents a rehabilitation technique based on a lower-limb exoskeleton integrated with a human–machine interface (HMI). HMI is used to record and process multimodal signals collected using a foot motor imagery (MI)-based brain–machine interface (BMI) and multichannel electromyographic (EMG) signals recorded from leg muscles. Current solutions of HMI-equipped rehabilitation assistive technologies tested under laboratory conditions demonstrated a great deal of success, but faced several difficulties caused by the limited accuracy of detecting MI electroencephalography (EEG) and the reliability of online control when executing a movement by patients dressed in an exoskeleton. In the case of lower-limb representation, there is still the problem of reliably distinguishing leg movement intentions and differentiating them in BMI systems. Targeting the design of a rehabilitation technique replicating the natural mode of motor control in exoskeleton walking patients, we have shown how the combined use of multimodal signals can improve the accuracy, performance, and reliability of HMI. The system was tested on healthy subjects operating the exoskeleton under different conditions. The study also resulted in algorithms of multimodal HMI data collection, processing, and classification. The developed system can analyze up to 15 signals simultaneously in real-time during a movement. Foot MI is extracted from EEG signals (seven channels) using the event-related (de)synchronization effect. Supplemented by EMG signals reflecting motor intention, the control system can initiate and differentiate the movement of the right and left legs with a high degree of reliability. The classification and control system permits one to work online when the exoskeleton is executing a movement.


I. INTRODUCTION
Today, exoskeletons are regarded as a powerful instrument for the clinical rehabilitation of patients with impaired lower-limb function (for a recent review, see Refs. [1], [2]). However, controlling the exoskeletons typically requires a physical action such as the push of a button, which is quite The associate editor coordinating the review of this manuscript and approving it for publication was Qichun Zhang . different from how normal motor actions are initiated. A more natural and intuitive way of interacting with exoskeletons and other neuroprosthetic devices is to use endogenous brain signals. This can be implemented using brain-machine interface (BMI) systems based on electroencephalographic (EEG) signals generated independently from external stimulation. Therefore, this approach allows the movement to be fully controlled by a subject. A typical example of such a BMI is a system based on sensorimotor rhythms such as motor imagery (MI) [3]. The two possible amplitude modulations of the sensorimotor rhythms include event-related desynchronization (ERD) and event-related synchronization (ERS). Motion-related tasks in both a real movement and MI are associated with a decrease in EEG power or ERD. As in the case of BMI-mediated upper-limb techniques, lowerlimb movement, which follows the activation of appropriate motor areas of the cortex, is believed to be able to accelerate the rehabilitation of patients by promoting neural plasticity [4]- [6]. Consequently, BMI systems based on the detection of ERD can be considered a promising technique to accelerate rehabilitation.
Although several studies demonstrate the efficiency of the BMI-assisted therapy of upper limbs [7]- [11], there is a lack of similar research on patients with lower-limb impairments. Similar to hand MI, BMI based on foot MI and integrated with a lower-limb exoskeleton would induce plastic changes in the affected brain areas and, hence, stimulate rehabilitation [1]. Anatomically, the left and right foot representation areas in the human sensorimotor cortex are located not only near the margo superior cerebri (the deep region inside the interhemispheric fissure) but also very close to each other [12]. That is why, in contrast to hand control, the classification of left and right foot imagery is quite difficult to implement. Very few studies discriminate between right or left foot imagery [13]- [17]. It should also be noted that all these studies were carried out under comfort conditions with long-term prior training. Such studies involved a large number of electrodes and used a long epoch in the EEG for classification. Unfortunately, these technologies cannot be implemented for the real-time control of an exoskeleton based rehabilitation device.
Moreover, even the use of general foot MI-based BMIs, which detect only foot MI without discriminating between the left or right side, creates difficulties in the real-time implementation of lower-limb exoskeleton control [1], [18]- [22]. Developing lower-limb MI-based BMIs for exoskeleton operation is considered difficult because of EEG signal contamination by artifacts from exoskeleton electronics, intensive body movements, and tonic muscle activity.
To enhance the reliability of a real-time EEG-based control system, BMI can be complemented with signals of other modalities, particularly with the electromyographic (EMG) recording of muscle activity. There have been several studies integrating the EMG and EEG modalities in the control system of upper-limb [8], [23], [24] and lower-limb [25]- [27] exoskeletons. This allows one to improve BMI system safety and to increase the degrees of freedom of assistive devices. The integration of EMG into an EEG-based control system of the lower-limb exoskeleton was crucially important not only for the quality of movement prediction but also for system adaptability to the long-term dynamics (evolution) of muscle activity during the rehabilitation process. For totally disabled subjects or subjects in the early stages of rehabilitation therapy with weak muscular activity, it was important to detect the user's movement intention by registering motor-related EEG signals and, subsequently, to provide feedback after execution of a corresponding movement by the exoskeleton. During the rehabilitation of patients with improving muscular activity, the reliable prediction of movement onset by EMG becomes more important for training more precise movements.
It is necessary to involve muscle activity (residual or appearing in the process of rehabilitation) in the control loop to form the correct stereotypes of limb movements. In this case, forming an imaginary reflex function (e.g., the activity of a local part of the motor cortex in the brain is not sufficient), the activity of the entire motor chain (somatosensory cortex, primary and secondary motor cortex, spinal cord motor neurons) has to be restored for the proper rehabilitation of motor functions.
Summarizing this brief review of HMI-equipped rehabilitation exoskeletons, we note that the major problems to be addressed include intention detection accuracy and control system performance in the integration of HMI into the exoskeleton control system. Specifically for the lower-limb exoskeletons, there are natural limitations in the separation of EEG patterns regarding the intention and execution of different leg actions because the corresponding representation areas are located quite close to each other. Other modalities, including leg EMG signals, can be helpful at certain stages of rehabilitation. Finally, the main challenge still to be addressed is the design of a reliable exoskeleton machine that can imitate the natural movement (e.g., step walking) of a patient dressed in an exoskeleton and naturally control it by brain signals using the MI of stepping legs (not the other limbs) with the synchronized detection and stimulation of leg muscles that should implement movement under healthy conditions. This provides very specific and physiologically correct activations of the brain areas synchronously with the muscle system, thus improving the rehabilitation procedure.
In this paper, we propose a novel multimodal humanmachine interface (mHMI) with integrated EEG and EMG modalities, which can provide real-time control of a lowerlimb exoskeleton. The exoskeleton follows a motor intention (leg lift) by decoding foot MI or motor execution. EEG modality was implemented via BMI based on ERD correlated with MI or motor execution. EMG modality was implemented via HMI based on the detection of muscular activity correlated with leg movement. The experiment, which led to development of the technique, was conducted in eight healthy subjects. The results showed a high accuracy rate in motion intention and execution classification tasks for the EEG and EMG modalities of our mHMI, respectively. Data analysis showed that the combination of EEG and EMG modalities can (i) improve the reliability of movement prediction by decreasing the false positive rate and (ii) enhance the positive detection rate of EEG-based classifications.

II. SYSTEM SETUP
The exoskeleton control system based on multimodal EEG-EMG HMI consists of the following parts: (1) an EEG and EMG signal recording module, (2) an EEG and EMG signal  processing and classifier module, (3) the control system of the exoskeleton, and (4) a lower-limb exoskeleton.
The signal recording module recorded the EEG signal during the MI by subjects and EMG signals during the leg movements. Furthermore, the EEG and EMG data were sent to the processing and classification module, where raw data were preprocessed by a feature extraction procedure. The classifier analyzed the preprocessed data to recognize motion intention. After motion intention is predicted, the control system sends the corresponding command to the exoskeleton, which finally executes the target movement. The scheme of our mHMIbased exoskeleton control system is given in Fig. 1.

A. EEG SIGNAL RECORDING MODULE
EEG signals were recorded using a certified NVX 52 amplifier (LLC ''Medical Computer Systems,'' Russia). Seven electrodes were used to record EEG (C5, C3, C1, Cz, C2, C4, C6) arranged according to the international 10-10 scheme ( Fig. 2). Such a scheme provides a denser coating of the interest area compared with other schemes (e.g., [10][11][12][13][14][15][16][17][18][19][20]. The reference electrode was placed on the ear lobe. The grounding electrode was placed on the forehead. The signal sampling rate was 500 Hz. Resistance under the electrodes did not exceed 10 k . The value of the automatically measured impedance of the skin contact (no more than 15 k ) was monitored to control contact during the application procedure.

B. EMG SIGNAL RECORDING MODULE
Disposable gel electrodes were attached to the wires of the NVX 52 amplifier. Two electrodes with one common reference for all channels were used for each EMG channel. Four EMG channels, which recorded the EMG of the musculus tensor fasciae latae (MTFL), musculus rectus femoris (MRF), musculus biceps femoris (MBF), and musculus gastrocnemius (MG), were used for each leg. Fig. 3 shows the location of the EMG electrodes on the leg. Electrode placement on the muscles, their alignment in accordance with fiber direction, and the distance between them were set according to the recommendations of the SENIAM project (surface EMG for the noninvasive assessment of muscles project) [28], [29].

C. LOWER-LIMB EXOSKELETON
The lower-limb exoskeleton shown in Fig. 4 was designed by the Scientific and Production Company ''MADIN'' (Nizhny Novgorod, Russia) in collaboration with the National Research Lobachevsky State University (Nizhny Novgorod, Russia). The exoskeleton was designed to help with rehabilitation training or walking assistance.
The exoskeleton is composed of the mechanical body and control and sensor systems. The mechanical body consists of a frame with a mounted control unit and battery, and two legs attached to the frame. Each leg consists of a femoral and knee drive connected by sliding units. The lengths of the legs were made adjustable to fit patients of different heights. Each leg ends with an insole with return springs. The drives consist of a reducer, electric motor, and analog angle sensor operated 84072 VOLUME 8, 2020 by a drive controller connected to the central control unit. The control unit commands the drives to perform the required movements while analyzing the readings from the sensors to control the accuracy of set movements. The exoskeleton can perform movements such as standing up, sitting down, and various types of walking. Motion initiation can be performed by control signals arising from the EMG and/or EEG signal classifiers. For safety reasons, in the case of incorrect classification results, the system control can always be intercepted from a button console.

A. DATA PREPROCESSING
The raw EEG and EMG data were filtered by bandpass filters with frequency ranges from 8 to 15 Hz and from 10 to 300 Hz, respectively. We also applied the Notch filter to remove the power voltage interference at 50 Hz.

B. EMG FEATURE EXTRACTION: ROOT MEAN SQUARE
After digital filtration, we extract the envelope of signals by root mean square (RMS) calculation: where x is an input EMG signal, N is the number of elements of the input data set, and t is time. Fig. 5 shows raw EMG data and the corresponding RMSs.

C. EEG FEATURE EXTRACTION: COMMON SPATIAL PATTERN FILTER
The discriminated features of the EEG signal in the case of MI were extracted using a common spatial pattern filter (CSP), which is considered to be one of the most effective filters in MI-based BMI technology [30], [31]. In analytical form, CSP can be described as where X is a multichannel EEG signal of M × N dimension, X CSP represents signal decomposition components (matrix of M × N dimension), and W is a decomposition matrix of M × M dimension. The decomposition matrix W is calculated according to the following algorithm. An intraclass average covariance matrix for the EEG signals from classes 1 and 2 is calculated as follows: where i indicates class 1 or class 2, N is the number of EEG signals from class i, and C l is the covariance matrix for signal X l . The average covariance matrix for the entire data is Then, the eigenvalues ( ) and the eigenvectors (U) of matrix C are obtained as follows: The whitening transformation matrix is P = −1/2 U T , and matrix C 1 is transformed into Diagonalization of matrix K gives where K is the diagonal matrix of the eigenvalues and U K is a matrix of eigenvectors (ordered in descending order of the corresponding eigenvalues). The decomposition matrix W has to satisfy the following conditions: where 1 , 2 are the diagonal matrixes of the C 1 , C 2 eigenvalues, respectively, and I is the unity matrix. Taking into account Eqs. (6)- (8), the resulting decomposition matrix W is determined as follows: The maximum dispersion values for the signal X CSP = W T X will be observed in the first k channels. For MI-based BMI technologies, k = 3 is commonly used. Thus, the feature vector is formed by the dispersion values in the first three channels of the signal X CSP .

D. LINEAR DISCRIMINANT ANALYSIS
Both the EMG-and EEG-based classifications were performed using a linear discriminant analysis (LDA) method. This method is a generalization of Fisher's linear discriminant and can be effective for the EMG/EEG classification problem because of its lower computational requirements. Similar to the statistical criterion in ANOVA, LDA aims to maximize the distance between the means of data classes and minimize the standard deviations of classes.
LDA implies a threshold classification according to where x represents input data to be classified, w is a normalized vector (defining the hyperplane separating classes), and c defines the threshold constant. Next, w and c are calculated as where is a class covariance (LDA assumes that the class covariances are identical), µ 0 , µ 1 are means of classes to be separated, and T is a certain threshold. LDA was implemented using the standard classify function of MATLAB. Fig. 6 shows examples of classification with the LDA method. It should be noted that here we performed EMGbased multiclass recognition (see Section IV B).

E. CLASSIFICATION PERFORMANCE METRICS
The obtained classification systems were evaluated using an accuracy value equal to the number of correct predictions divided by the total number of predictions.
Statistical measures of the performance of the binary classification test consisted of the true positive rate (TPR), false positive rate (FPR), true negative rate (TNR), and false negative rate (FNR). These performance metrics were defined as follows: (12) where TP is the number of correctly classified leg movement trials, P is the total number of leg movement trials, FN is the number of incorrectly classified leg movement trials, FP is the number of incorrectly classified rest trials, N is the total number of rest trials, and TN is the number of correctly classified rest trials.
Another measure of performance was balanced accuracy (BA), which was used to evaluate the performances obtained from the two types of signals (EEG/EMG) and their combinations (AND/OR) together. BA was defined as follows:

F. ERD ANALYSIS
To evaluate the degree of ERD/ERS of sensorimotor rhythm during MI, patterns corresponding to the resting task were taken as the reference state [32]. The EEG signals were spatially filtered using the surface Laplacian for all the channels. Then, the power spectral density was constructed for each signal with a step of 1 Hz, and ERD was calculated as the difference in the signal powers during MI and the rest signal, which was divided by the signal power corresponding to the rest task.

IV. EXPERIMENTS
For experimental purposes, we recruited eight healthy volunteers of either sex aged from 20 to 27. The study complied with the Helsinki declaration adopted in June 1964 (Helsinki, Finland) and revised in October 2000 (Edinburgh, Scotland). The Ethics Committee of the Lobachevsky State University of Nizhny Novgorod approved the experimental procedure (protocol No. 28 of January 1, 2019). All participants gave their written consent. All subjects previously had no experience with BMI. Before the experiments, the subjects were interviewed to find out which of their legs was dominant. After application of the EMG and EEG electrodes, each subject put on the exoskeleton and took a neutral stance, leaning on crutches. The exoskeleton was adjusted to fit each subject. Throughout the experimental session, the subject remained in the exoskeleton. The duration of the experiment did not exceed 90 min. The experiment consisted of two parts and was carried out by an online procedure. The first part consisted of training and testing the performance of HMI to control the exoskeleton using only foot MI. The second  part consisted of testing with real foot movements. During all the experiments, the EEG and EMG signals were collected simultaneously from 15 channels (seven EEG channels and eight EMG channels).

A. HMI BASED ON MI
To develop the scheme of the EEG-based HMI for exoskeleton control, we adapted the approach proposed in our previous work [33]- [35].
To improve the classification accuracy in our study, we used BMI based on the MI of only one dominant foot. Throughout the training and testing sessions, the subject stood wearing the exoskeleton. During classifier training, the subjects performed one of two commands: virtual imaging of movement of the dominant leg or rest, when the subject had to concentrate on his/her breathing. Commands were given to the subjects via a light-emitting diode display. The examples of images are shown in Fig. 7. An image of a ''right/left'' arrow corresponded to the MI of leg movement, and a cross corresponded to the rest state. Between the commands, a gray monitor without any images was shown to the subjects during the intervals of rest and the preparation stage. Each 4-s command was presented 5 times. The interstimulus interval was 3 s (empty screen). The stimuli were presented randomly. The classifier training duration amounted to 1 min and 10 s. The scheme of the training session is shown in Fig. 8a.
When training of the classifier was completed, each subject had three testing sessions controlling the exoskeleton. Each command was repeated 5 times during a testing session. The duration of the command was 4 s, and the interstimulus interval was increased to 10 s. The commands were presented randomly. When testing HMI, the classifier analyzed EEG every 500 ms, and the results of mental task recognition were shown to the operator via visual feedback every 500 ms: a green scale beginning in the circle in the screen center, where subjects were to fix their eyes, filled down to the edge of the screen if the classifier recognized the task to be in agreement with the given command; the scale would stop filling when another task was recognized (Figs. 7 and 8b). If HMI obtained the correct classification result for 4 s of MI, the exoskeleton made a step by moving the corresponding leg. In the case of incorrect classification, the exoskeleton did not perform any action. The scheme of the testing session is shown in Fig. 8b.

B. HMI BASED ON EMG
To train the classifier, one of the three movement classes was randomly presented to the test subject: ''attempt to step with the left foot'' (LL class), ''attempt to step with the right foot'' (RL class), and ''neutral position'' (rest). As in the previous case, there were corresponding images shown to the subjects (Fig. 6). Each class lasted 3 s, and the interval between the classes was 10 s. The duration of the training session was 220 s.
When classifier training was completed, each subject had three test sessions. During testing, feedback with the subject was carried out through the exoskeleton's reaction. In the case of correct classification, the exoskeleton took a step with the corresponding foot. In the case of incorrect classification, the exoskeleton did not perform any action. Fig. 9 shows the results of the real-time control of the exoskeleton by BMI based only on foot MI. Classification accuracy was calculated for all subjects in each of three tests and then averaged for all the tests. This classification was performed to discriminate between the imagery of foot movement or rest. Note that the classification accuracy under ideal conditions could reach 100%, with a chance level of 50%. In our experiments, the classification accuracy was relatively high, equal to 78.3% on average (60%-100%; SD 12.24%) for all subjects. The greatest average classification accuracy for all attempts was observed in Subject 3 (86.7%), and Subject 4 showed the highest accuracy in the first test (100%).

A. REAL-TIME BMI CONTROL OF AN EXOSKELETON BASED ON LOWER-LIMB MI
The peak values of ERD in the band 8-16 Hz corresponded to the µ (6-12 Hz) and low β rhythm (13-17 Hz) during sessions of MI obtained for each electrode for each subject.
The average values of µ-ERD for all subjects (for one subject, VOLUME 8, 2020 we excluded one outlier test of MI) in different electrodes are indicated in Fig. 10. The ERD levels in different electrodes for all subjects separately are shown in the Supplementary material in Fig. 1. Fig. 10 shows that, on average, all subjects generated larger ERD values in the Cz channel (e.g., vertex area) as expected, because lower-limb movements are represented in this cortical region. The averaged ERD values for the other channels are close to ERD on Cz. The presence of such background EEG activity in the areas of the hands can be explained by the fact that subjects wearing the exoskeleton used their hands to maintain balance. Thus, the contribution of EEG patterns corresponding to the hand movements to the foot MI of the exoskeleton wearer was considerable. We tried to minimize such interference. We trained and tested the system in the same position while the subject maintained the standing position in the exoskeleton. It is important to note that the ERD levels on electrodes C3 and C4, corresponding to the hand's areas in the cortex, for foot MI (Fig. 10) was twice less, respectively to the hand MI in our previous study [34]. Note also that here we do not focus on the classification of foot MI [13], [15], [16] but rather on the ability and performance of online control of the exoskeleton using foot MI under real conditions. Examples of the EEG topographies of ERD following foot MI are shown in the Supplementary material in Fig. 3.

B. REAL-TIME EEG-BASED BMI CONTROL OF AN EXOSKELETON BASED ON LOWER-LIMB MOTOR EXECUTION
In experiments with motor execution, BMI detected ERD in the EEG signals when subjects lifted their right or left leg. The classification results for this case are shown in Fig. 11. We obtained classification accuracy for two (both legs together without discriminating between the left or right side, or rest) (Fig. 11a) and three (right/left leg or rest) (Fig. 11b) classes. The accuracy in the case of two classes, on average, for all subjects was 78.13% (70.67%-86.67%; SD 17.37%), which is almost the same as for MI. The classification accuracy for three classes (right/left legs and rest) reached quite a low value equal to 51.31%, on average (33%-66.3%; SD 17.14%), for all subjects. This value, however, was larger than a chance level of 33.3%, but still not acceptable to control the exoskeleton. The greatest classification accuracy was observed in Subject 2, on average, for all attempts for two classes (86.7%) and for three classes (66.3%). For Subject 6, the accuracy for three classes was close to a random value of 33.3%. Fig. 12 shows the average values of ERD during real movement of the left (a) and right (b) leg for all subjects in different electrodes averaged for all tests. The ERD levels in different electrodes for all subjects separately are shown in the Supplementary material in Fig. 2. Unlike foot MI, in the case of real movement in the exoskeleton, the largest ERD values were observed for channels C3 and C4 and not for the Cz channel. Channels C3 and C4 corresponded to the hand representation areas in the sensorimotor cortex. Desynchronization of these areas was more significant than in the case of MI because, although the exoskeleton lifted the leg, subjects had to lean on the crutches to maintain balance. That is why we obtained poor classification accuracy for the real movement of different legs.

C. REAL-TIME EMG-BASED HMI CONTROL OF AN EXOSKELETON BASED ON LOWER-LIMB MOTOR EXECUTION
When the exoskeleton was turned on, the subject's legs were fixed. Therefore, an attempt to make a step would create isometric tension in the muscles. Moreover, even just standing in the exoskeleton did not allow full relaxation of the leg muscles. Thus, the classification of EMG patterns under such conditions can be difficult. Using all the EMG channels (four for one leg), we gained an accuracy of classification ranging from 61% to 100% for different subjects (Fig. 13, ''ALL 4''). On the basis of our previous study, such a large scatter can be explained by differences in the anatomical features and muscle coordination of different people [36]. The average accuracy was 83.4%, SD = 14.2% (Fig. 14, ''ALL 4'').
We hypothesized that muscles differ from each other in information value in the context of EMG classification and, accordingly, made different contributions to the accuracy. However, classification based on recording the activity of one muscle (on each leg) did not reveal a leader or an outsider (Fig. 13). For example, in Subject 2, the classification based on MG showed the best accuracy, whereas in subject 8, MRF was the most informative.
The classification results averaged over all subjects using full and reduced sets of EMG channels (Fig. 14) also indicated approximately equal information values of different muscles. The best average accuracy for all subjects was 85.8% (SD = 13.3) with a set of EMG channels, including MTFL, MRF, and MBF (''TFL + RF + BF'' in Fig. 14). However, the differences in the results based on different sets were not significant.
Thus, the average accuracy of the EMG-based classification, as expected, was higher than in the case of EEG control. However, considering that the exoskeleton was intended for  Accuracy of the classification of the three motion classes averaged for all subjects operating the exoskeleton using different sets of EMG channels. The notation of the EMG channels is similar to that of Fig. 13. Average values and standard errors are indicated.
patients with motor disorders, the main problem was the possibility of joint multimodal classification.

D. OFFLINE ANALYSIS OF MULTIMODAL EEG-EMG HMI PERFORMANCE
Unlike the case of detecting foot MI, we can predict the attempt of real foot movement using combinations of the EEG and EMG signals. We developed two protocols for combining EEG and EMG: (i) HMI based on extracting CSP features with subsequent LDA classification (Fig. 15a) and (ii) HMI based on separate feature extraction and classification, the results of which were combined by logical operators ''AND'' and ''OR'' (Fig. 15b). Here we used EEG-and EMG-based classification for two classes (1: foot movement execution without discriminating between the left or right side; 2: rest) VOLUME 8, 2020 because of the low EEG-based classification accuracy value for three classes.
The EMG + EEG classification based on CSP features with subsequent LDA classification (Fig. 15a) showed an accuracy of 80% (SD = 13%), which can be considered as a kind of ''averaging'' of the EEG-based (78.13% for two classes) and the EMG-based (89%, SD = 10% for discriminating two classes) accuracies. However, in the case of impaired motor activity (muscle spasm, atrophy, etc.) that occurs as a result of pathological processes at the level of the spinal cord (trauma) or the motor cortex (stroke), the EMG signals can be significantly less informative.
To study the capabilities of multimodal HMI in the case of patients with motor impairment, we simulated the distortion of EMG signals by superimposing white Gaussian noise (σ = 0.0012) on the initial EMG signals of healthy subjects. The classification accuracy of noisy EMG (nEMG) signals, on average, would equal 59% (SD = 7.9%). Further, nEMG signals were supplemented with the original EEG signals. The accuracy of multimodal nEMG + EEG classification was, on average (for eight subjects), 70% (SD = 6.2%), which was lower than the results obtained in the case of real movement classification based only on EEG (78.13%). Thus, it was shown that a multimodal interface-based on a single classifier shows accuracy close to average accuracy for EMG and EEG.
For further analysis, we calculated additional metrics of classification performance such as TPR, FNR, TNR, FPR (12), and BA (13). The obtained metrics are summarized in Table 1 and visualized in Fig. 16. Despite the fact that the EEG-based control system showed relatively high accuracy, it led to much more high false positives (0.42) than that of an EMG-based system (0.04).
Furthermore, we studied the possibility of combining the results of independent EEG and EMG classification by the logical operators ''AND'' and ''OR.'' Condition ''OR'' meant that we predicted movement execution if either EEG-  or EMG-based analysis or both predicted a movement. Condition ''AND'' meant that we predicted movement execution if both the EEG-and EMG-based analyses predicted the movement. We found that the EMG signal and ''OR'' combination were the best types of signals in the classification of movement trial. The combination ''AND'' and the EMG signal were better than the EEG and ''OR'' types of signals in the classification of the rest state. For error type I (false positive), the combination ''AND'' and the EMG signal were the most accurate, and these types of signals were significantly better than the EEG and combination ''OR'' types of signals (AND/EMG p > 0.5; AND/EEG p < 0.001; AND/OR p < 0.001). For error type II (false negative), the combination ''OR'' was the most accurate, and the combination ''AND'' was the least accurate; the EEG signal was better than the EMG signal (OR/EMG p < 0.05; OR/EEG p < 0.001; OR/AND p < 0.001).

VI. DISCUSSION
In our study, we simulated realistic conditions for subjects controlling an exoskeleton integrated with multimodal HMI.
We proposed a mHMI control system based on foot MI or foot motor execution (e.g., real leg movement). All experiments were carried out in the online procedure when subjects were wearing the exoskeleton and controlled it using mHMI. We also showed that a small number of EEG electrodes (seven electrodes) is sufficient to provide effective online control.
The control of assistive devices by EEG or EMG signals separately can have major applications in rehabilitation. For example, the reliable detection of movement intention by motor-related EEG signals is crucially important for totally disabled subjects or subjects at the early stages of rehabilitation therapy with very weak muscular activity. EMG-based detection of an attempt to move would be more appropriate for subjects with residual muscle activity or subjects at a later state of therapy. Our results eventually demonstrated how a high accuracy rate can be achieved in both types of EEG-and EMG-based HMI for control of a lower-limb exoskeleton. The classification accuracy of EEG-based HMI in the case of foot MI for two classes was approximately 78%. Note that the classification accuracies for MI and actual movement execution were very close. This phenomenon can be explained by the fact that the exoskeleton facilitated kinesthetic imagery, which is believed to be more effective for BMI than visual imagery [37]. The accuracy of EMG-based control was 89%/83.4% for two/three classes.
However, there are circumstances when movement prediction based on EEG or EMG has limited performance. For example, EMG signals are not a reliable source for movement onset detection for patients with spasm. In turn, the EEG-based systems showed a high rate of false positives. Therefore, the combination of multimodal data should be especially relevant. It has already been shown in other studies [8], [24], [26], [38]- [40] how such combinations of different physiological data can improve the performance of detection of a subject's intentions.
In the case of simple HMI based on extracting CSP features from an EEG + EMG combination with subsequent LDA classification, we obtained an accuracy equal to 80%. In the case of EEG and noisy EMG signals, the classification accuracy was 70%. In both cases, the EEG + EMG combination led to ''averaging'' of the EEG-and EMG-based accuracies.
In the offline procedure, we investigated the performance of different logical combinations of the EEG and EMG modalities. We showed that the multimodal approach for detecting a movement attempt of an exoskeleton operator can either (i) improve the reliability of movement prediction by decreasing the FPR or (ii) enhance the positive detection rate. In particular, we gained a minimal FPR of 0.022 for the EEG AND EMG combination. False positive errors are the most dangerous errors for an operator of a lower-limb exoskeleton because they can provoke unexpected movements of the exoskeleton, thus risking the possibility of the subject losing balance. Thus, the EEG AND EMG combination can be used in cases when safety is prioritized. The maximal TPR was 0.996 for EEG OR EMG. The high TPR can be useful for encouraging patients with weak muscular activity (early rehabilitation stage) if there are no risks associated with the exoskeleton movement. Interestingly, a similar approach was recently investigated by Kirchner et al. [24] to predict selfinitiated hand movement onset by the offline analysis of EEG and EMG activities. In this study, all classification modalities had high performance in the range of 0.88-0.94 BA. They also studied different combinations of EEG and EMG analysis. Similar to our results, they showed that the best result of prediction movement onset was achieved using the ''OR'' combination and EMG as a single modality, and the ''AND'' combination can enhance the reliability of movement detection (i.e., decrease the FPR).
Finally, we believe that our findings on a combination of EEG and EMG signals can be further implemented in clinical rehabilitation protocols of robotic exoskeleton use with respect to individual demand, neuromuscular disorder, and state of rehabilitation therapy. Future research of the mHMI integrated exoskeleton control system can be focused on the design of adaptive feedback during the rehabilitation procedure. Specifically, this feedback can regulate motor power in collaboration with muscle activation during movement execution at the later stages of rehabilitation when muscular strength has recovered sufficiently. In other words, the machine would adaptively doze its power, thus encouraging patients to take steps instead of carrying them as passengers. From the brain side, another challenge is to use transracial magnetic stimulation, stimulating a particular representation area to improve control signal conduction from the center to the periphery.