Brain-Inspired Recognition System Based on Multimodal In-Memory Computing Framework for Edge AI

With the rapid development of edge devices, the deployment of AI on these devices has become a focal point for researchers. Moreover, the emergence of multimodal neural networks has facilitated the development and application of brain-inspired cognitive and recognition systems. However, the limited storage and computing resources of edge devices, along with the robustness issues of computational circuits, pose significant challenges in implementing AI systems with brain-inspired recognition capabilities on edge devices. To address this, we propose a memristor-based brain-inspired recognition (MBR) system, which can mimic the brain’s information processing mechanism without additional cross-modal processing, making its behavior more akin to human responses. In addition, the proposed MBR system is implemented using the multimodal in-memory computing (IMC) framework and validated the robustness and effectiveness of the proposed system through simulation analyzing. Furthermore, the multimodal sentiment analysis (MSA) and emotion recognition in conversation (ERC) tasks are implemented on the MBR system with only one single training process. The results demonstrate the proposed system achieves superior performance compared to most existing baseline methods. Lastly, since the MBR system relies on memristor-based matrix multiplication, it emerges as one of the promising solutions for edge-based brain-inspired recognition applications.