Decoding with Purpose: Improving Image Reconstruction from fMRI with Multitask Learning

Advancements in machine learning and computer vision have enabled medical researchers to find a breakthrough in computational neuroscience at a pace never seen before. However, such methods and tools remain inaccessible due to the semantic gap between the domains. One such neural information processing task, decoding fMRI to generate perceived images and analyze visual cortex function, has emerged as an exciting research topic. Existing works use multiple linear decoders or complete end-to-end deep networks for this task. Here we present a novel approach to neural network regression training for decoding tasks and compare the image reconstruction results with existing methods. First, we train an image generator model, which takes image features as input for image reconstruction, on random images which are not part of (fMRI→image) training pairs. The main highlight of the proposed system is our decoder with custom loss function and purpose-based training, which uses the discriminator of our image generator model to improve the image generation quality from decoded image features. Upon testing, our proposed approach with a correlation winning percentage of ∼81% manages to surpass the existing methods (∼78%) in reconstructing test images with better colour and shape reproduction. The approach proposed in this study provides better generalizability over existing decoder-based approaches and more control over training than end-to-end approaches, which can be extended to other such neural information processing tasks.