Semantic Decoding of Natural Image Stimuli in Functional Magnetic Resonance Imaging Based on Deep Learning

Decoding environmental inputs and cognitive states from brain activities is fundamental challenge in neuroscience. In recent years, deep learning has made significant strides in decoding task-state fMRI images of the human brain. However, progress has been affected by limitations such as the low signal-to-noise ratio of fMRI and the signal overlap caused by the hemodynamic response. Moreover, there has been no research on semantic decoding of natural image stimuli using whole-brain fMRI as input so far, while semantic information is actually distributed throughout the whole brain. Therefore, this study aims to build an end-to-end whole-brain fMRI natural image stimulus semantic decoder based on a deep learning model under natural image stimuli. This decoder can map fMRI brain activities into the processed image feature space, so as to utilize the mapped fMRI features to achieve and retrieval tasks. The correct rate of the retrieved pictures being completely consistent with the original stimulus pictures can reach up to 78.8% for a single subject. If pictures that are completely consistent in retrieval or have overlapping semantic information are considered simultaneously, the average retrieval accuracy of this model across all subjects can reach over 90%.