Brain Visual Stimulus Reconstruction Using a Three-Stage Multilevel Deep Fusion Model

Currently, the field of brain visual stimulus decoding from functionally synthesized MRI images has gained increaseing attention from researchers. However, challenges such as the scarcity of fMRI data samples and the low signal-to-noise ratio of fMRI data have resulted in issues related to pixel-level reconstructed images. These problems include low clarity, structural confusion, and unclear semantics. To address these challenges, this paper employs a three-stage multiscale deep fusion model (TMDFM). The model follows a three-stage training process that extracts low-level feature maps from depth images, high-level feature information from original images, and then combines them to obtain improved feature representations. Additionally, this paper incorporates a random shifting module, a dual attention module, and a multiscale feature fusion module to enhance image generation. In the context of the Horikawa17 dataset, quantitative comparisons using paired SSIM and human perceptual similarity achieved average scores of 76.0% and 92.0%, respectively, outperforming other methods. Moreover, this model demonstrates strong generalization capabilities by delivering impressive results on a handwritten dataset. Finally, this paper explores the contributions of different brain regions to decoding results and conducts several ablation experiments.