Fusing sEMG and EEG to Increase the Robustness of Hand Motion Recognition Using Functional Connectivity and GCN

Surface electromyogram (sEMG) is widely used in active rehabilitation control for stroke patients. However, the accuracy of movement recognition using sEMG signals is affected by abnormal states such as muscular fatigue and muscle weakness. In this article, a multimodal fusion strategy of electroencephalogram (EEG) and sEMG is proposed to improve the accuracy and robustness of hand motion recognition. It is an end-to-end approach based on a graph theory, in which the temporal signals of EEG and sEMG are considered as the features of nodes, and the functional connectivity is considered as the weights of edges. Four topologies, namely, 2EnMe, 2EwMe, 5EnMe, and 5EwMe, are proposed, and two standardization methods are tested for each topology. Then, three functional connectivity methods are investigated, namely, Pearson coefficient, mutual information, and coherence. Ten rounds of fivefold cross validation show that graph convolutional network (GCN)-2EnMe with the Pearson coefficient and min–max standardization is the best fusion model. At the fatigue levels of 0% and 30%, the achieved average accuracies are, respectively, 93.86% and 91.23%, which are significantly higher than those when using a parallel fusion method and a single-modality model. Moreover, the accuracy decrease ratio (ADR) of GCN-2EnMe is 2.80%, which is considerably better than that of a convolutional neural network (CNN) with parallel fusion (7.87%) and a CNN with single-modality sEMG (11.15%). The results show that the proposed novel EEG and sEMG fusion method has the potential to improve the accuracy and reliability of active control for stroke rehabilitation.