Semantics-guided hierarchical feature encoding generative adversarial network for natural image reconstruction from brain activities

The use of deep learning methods to decode visual perception images from brain activity recorded by fMRI has received a lot of attention. However, limited fMRI data make the task of visual reconstruction challenging. Inspired by hierarchical encoding of the visual cortex and the theory of brain homology with convolutional neural networks (CNNs), we propose a novel neural decoding model called hierarchical semantic generative adversarial network (HS-GAN). Specifically, we use CNN-based image encoder to extract hierarchical and semantic features of visually stimulus images. Then a neural decoder is used to decode hierarchical and semantic features from fMRI. In order to take full advantage of the information from different visual cortexes, we construct a generator with self-attention modules and skip connections to fuse the image features of different layers. In model training, adversarial learning is introduced to realize more natural image reconstruction. Compared to existing advanced methods, our method significantly improves the naturalness and fidelity of reconstructed images.