Functional Alignment-Auxiliary Generative Adversarial Network-Based Visual Stimuli Reconstruction via Multi-Subject fMRI

Functional Magnetic Resonance Imaging (fMRI) provides more precise spatial and temporal information to reconstruct stimulus images than other technologies that can be used to measure the human brain’s neural responses. The fMRI scans, however, generally show heterogeneity among different subjects. The majority of the existing methods aim primarily at mining correlations between stimuli and evoked brain activity, disregarding the heterogeneity among subjects. Therefore, this heterogeneity will impair the reliability and applicability of multi-subject decoding results, leading to sub-optimal results. The present paper proposes the functional alignment-auxiliary generative adversarial network (FAA-GAN) as a novel multi-subject approach for visual image reconstruction that employs functional alignment to alleviate the heterogeneity between subjects. Our proposed FAA-GAN includes three key components: 1) a generative adversarial network (GAN) module for reconstructing visual stimuli, which consists of a visual image encoder as the generator that uses a nonlinear network to convert stimuli images into an implicit representation and a discriminator that generates the images comparable to the original images in detail; 2) a multi-subject functional alignment module, which is used to precisely align the individual fMRI response space of each subject in a common space to reduce the heterogeneity among different subjects; and 3) a cross-modal hashing retrieval module used for similarity retrieval of two modalities of data, i.e., the visual images and the evoked brain responses. Experiments on real-world datasets show that our FAA-GAN method does better than other state-of-the-art deep learning-based reconstruction methods with fMRI.