Speech2EEG: Leveraging Pretrained Speech Model for EEG Signal Recognition

Identifying meaningful brain activities is critical in brain-computer interface (BCI) applications. Recently, an increasing number of neural network approaches have been proposed to recognize EEG signals. However, these approaches depend heavily on using complex network structures to improve the performance of EEG recognition and suffer from the deficit of training data. Inspired by the waveform characteristics and processing methods shared between EEG and speech signals, we propose Speech2EEG, a novel EEG recognition method that leverages pretrained speech features to improve the accuracy of EEG recognition. Specifically, a pretrained speech processing model is adapted to the EEG domain to extract multichannel temporal embeddings. Then, several aggregation methods, including the weighted average, channelwise aggregation, and channel-and-depthwise aggregation, are implemented to exploit and integrate the multichannel temporal embeddings. Finally, a classification network is used to predict EEG categories based on the integrated features. Our work is the first to explore the use of pretrained speech models for EEG signal analysis as well as the effective ways to integrate the multichannel temporal embeddings from the EEG signal. Extensive experimental results suggest that the proposed Speech2EEG method achieves state-of-the-art performance on two challenging motor imagery (MI) datasets, the BCI IV-2a and BCI IV-2b datasets, with accuracies of
${89}.{5}\%$
and
${84}.{07}\%$
, respectively. Visualization analysis of the multichannel temporal embeddings show that the Speech2EEG architecture can capture useful patterns related to MI categories, which can provide a novel solution for subsequent research under the constraints of a limited dataset scale.