Motor Imagery Brain-Computer Interaction Using Different Modalities of Action Observation

Current research has examined the impact of different modalities of action observation (AO) on motor imagery (MI) performance from the perspective of event-related desynchronization (ERD). However, in practical brain-computer interface (BCI) applications, the classification performance of MI is crucial. This article conducted experiments in which hand movements are observed during MI using different modalities: image, video, and entity. By analyzing ERD performance and classification performance of MI electroencephalogram (MI-EEG) signals, we investigate the impact of different modalities of AO on MI performance. To assess the classification performance, this article proposes a novel multiscale separable convolutional neural network (MSSCNN). With fewer parameters, it extracts broader spatiotemporal features from MI-EEG signals at multiple scales, which improves the accuracy of the classification results. The experimental results illustrate that MI + AO-Entity outperforms MI + AO-Video and MI + AO-Image in ERD performance and classification performance, with their average classification accuracies reaching 91.97%, 88.29%, and 83.75%. In addition, we designed a soft robotic glove control system based on a motor imagery brain-computer interface (MI-BCI) and conducted online experiments. The results suggest that different observation modalities can effectively improve MI classification accuracy and enhance MI performance to varying degrees.