Framework for Emotion Recognition Using Cross-Modal Transformers With Non-Contact Multimodal Signals Aiming Clinical Service Support

In clinical communication, it is believed that accurately understanding and appropriately responding to patients’ emotions contributes to treatment effectiveness and patient satisfaction. Recently, multimodal approaches that integrate various modalities such as speech, text, and physiological signals have gained attention for emotion estimation. However, the application of emotion recognition (ER) technology in clinical settings has not been thoroughly explored, particularly in terms of utilizing non-contact measurement techniques to reduce patient burden. Furthermore, there is limited research on quantitatively evaluating physicians’ ER abilities and comparing them with existing ER methods. This study aims to propose a multimodal ER framework using non-contact measurement techniques and validate its effectiveness by comparing it with the emotion prediction accuracy of experienced physicians. The results demonstrated that the proposed non-contact multimodal approach outperformed physicians in ER accuracy. While physicians’ empathetic abilities have traditionally been considered high, integrating multiple modalities was shown to surpass the recognition accuracy of unimodal approaches and human physicians. Moreover, the ability to obtain emotion-related data non-invasively enables advanced emotion estimation while reducing physical and psychological burdens on patients, highlighting the potential for clinical applications. These findings suggest a new method for supporting physicians’ ER in clinical settings, offering a means to reduce the risk of fatigue associated with empathy.