MI-MAMI: Multisensory Integration Model Inspired by the Macro and Micro Mechanisms of the Human Brain

The interaction between information from different senses is crucial for multisensory integration. Without the interaction, traditional deep learning methods exhibit inferior performance in the integration. On the contrary, the human brain has an inherently remarkable ability in this area, which stems from the macro and micro brain networks. Based on this superiority, we propose a brain-inspired model called MI-MAMI (Multisensory Integration model inspired by the MAcro and MIcro mechanisms). Aligning with the macro brain network that consists of unisensory pathways and integration regions, a similar framework of the MI-MAMI is established. The MI-MAMI incorporates unisensory processing streams and an integration block, enabling the comprehensive integration of multisensory information. In addition, inspired by the diverse neurons and the synaptic connections found in integration regions, we innovatively design weight constraints to regulate information transmission among neurons. Leveraging these mechanisms at both the macro and micro levels, our model is biologically plausible and enables more effective interaction and integration. To validate the performance of the MI-MAMI model, we conducted a case study focusing on multisensory emotion recognition. The results show that our model surpasses state-of-the-art brain-inspired baselines on RAVDESS and eNTERFACE’05 datasets, showcasing the potential of brain-inspired methods for advancing multisensory integration.