Spatio-Spectral Feature Representation for Motor Imagery Classification Using Convolutional Neural Networks

Convolutional neural networks (CNNs) have recently been applied to electroencephalogram (EEG)-based brain–computer interfaces (BCIs). EEG is a noninvasive neuroimaging technique, which can be used to decode user intentions. Because the feature space of EEG data is highly dimensional and signal patterns are specific to the subject, appropriate methods for feature representation are required to enhance the decoding accuracy of the CNN model. Furthermore, neural changes exhibit high variability between sessions, subjects within a single session, and trials within a single subject, resulting in major issues during the modeling stage. In addition, there are many subject-dependent factors, such as frequency ranges, time intervals, and spatial locations at which the signal occurs, which prevent the derivation of a robust model that can achieve the parameterization of these factors for a wide range of subjects. However, previous studies did not attempt to preserve the multivariate structure and dependencies of the feature space. In this study, we propose a method to generate a spatiospectral feature representation that can preserve the multivariate information of EEG data. Specifically, 3-D feature maps were constructed by combining subject-optimized and subject-independent spectral filters and by stacking the filtered data into tensors. In addition, a layer-wise decomposition model was implemented using our 3-D-CNN framework to secure reliable classification results on a single-trial basis. The average accuracies of the proposed model were 87.15% (±7.31), 75.85% (±12.80), and 70.37% (±17.09) for the BCI competition data sets IV_2a, IV_2b, and OpenBMI data, respectively. These results are better than those obtained by state-of-the-art techniques, and the decomposition model obtained the relevance scores for neurophysiologically plausible electrode channels and frequency domains, confirming the validity of the proposed approach.