Patient's Emotion Computing Under Robot Vision

With the development of artificial intelligence technology, the research of hospital ward inspection robots has received strong support from hospitals, improving the intelligence of ward inspection. This paper designs a multimodal video emotion computing model based on the dual buffering mechanism from the perspective of hospital inspection ward robots. This model can adjust the weight of the model's voice modality based on the decibel value of the environment in a hospital environment, and recognize the emotions of patients in a bimodal manner of voice and face, assisting medical staff in decision-making. This model separates voice and face features from videos in real-time and synchronously buffers them into two queues, forming a voice segment sequence queue and a face sequence queue. In order to extract deep temporal and spatial features of voice segments, MFCC was used to extract primary features, and LSTM was used to extract deep temporal features. Simultaneously, convert voice information into spectrograms, extract spatial features from spectrograms using ResNet50, and extract features from face information using ResNet50. The model has strong feature extraction ability.Then, the extracted deep features were combined with self attention mechanism, cross attention mechanism, and multi head attention mechanism to focus on the correlation information within and between modalities. And fused the deep features through attention mechanism twice, and finally used softmax for emotion classification. After experiments, the accuracy of the model has been improved compared to the single modal classical model and the multimodal classical model. At the same time, ablation experiments have also verified the efficiency of the multimodal model in this paper.