Linking Brain Signals to Visual Concepts: CLIP based knowledge transfer for EEG Decoding and visual stimuli reconstruction

Decoding visual representations from human brain activity has emerged as a thriving research domain, particularly in the context of brain-computer interfaces. This study introduces a novel approach using a convolutional neural network (CNN) to classify images from the ImageNet dataset, leveraging electroencephalography (EEG) recordings. We collected EEG data from 6 subjects, each viewing 50 images across 40 distinct semantic categories. These EEG signals were transformed into spectrograms, serving as the input for training our CNN. A unique aspect of our model is the incorporation of knowledge distillation from a pre-trained image classification teacher network. This approach enabled our model to achieve a top-5 accuracy of 80%, notably surpassing a plain CNN baseline. Furthermore, we integrated an image reconstruction pipeline founded on pre-trained latent diffusion models. This innovative concatenation not only decodes images from brain activity but also provides a plausible reconstruction, facilitating rapid and subject-specific feedback experiments. Our work thus represents a significant advancement in the field, bridging the gap between neural signals and visual perception.