A Temporal Envelope-based Speech Reconstruction Approach with EEG Signals during Speech Imagery

This work studied a brain-computer interface (BCI) system for speech synthesis based on imagined electroencephalography (EEG). The system incorporated a vocoder decomposition layer, a Gaussian process regression (GPR) layer and a vocoder synthesis layer, and was evaluated with speech recordings and imagined EEG signals from a public dataset (i.e., KARAONE). The raw speech signals were decomposed into envelopes in 12 frequency bands. Imagined EEG features were projected to each speech envelope by GPR, then the projected envelopes were used for speech reconstruction. With a cross-subject evaluation scheme, the similarity between the raw and projected envelopes achieved an average normalized covariance of 0.57, and the short-term objective intelligibility measurement between the raw and reconstructed speech yielded an average value of 0.70. Results in this work suggested the potential in developing a BCI-based communication with intelligible speech reconstruction.