VTVBrain: A Two-stage Brain Encoding Model for Decoding Key Neural Responses in Multimodal Contexts

In the field of cognitive neuroscience, understanding how the brain processes multimodal complex stimuli is a long-standing and complex challenge. In this study, we propose a novel two-stage brain coding model called "VTVBrain" that focuses on decoding key neural responses in multimodal environments. In the first stage, we built a high-dimensional multimodal latent space pre-training model using an attentional mechanism-based variational autoencoder, aiming to capture and encode the main perceptual processes of observers when faced with multimodal stimuli integrating textual and visual information. In the second stage, the Versatile Diffusion model is utilized for image reconstruction of the first stage primary representational images for high-level perception. The proposed model further refines the latent space representation approach, explores the relationship between the brain’s neural responses and complex natural scenes, and provides insight into how the brain integrates and processes information from different senses at a higher level. The introduction of the VTVBrain model not only brings a new perspective to the field of neural decoding, but also has a great potential for application in the development of brain-like artificial intelligence and future cognitive neuroscience research. The introduction of the VTVBrain model not only brings a new perspective to the field of neural decoding, but also has great potential for the development of brain-like artificial intelligence and future cognitive neuroscience research.