A Metaverse Emotion Mapping System with an AIoT Facial Expression Recognition Device

Recently, there has been a lot of interest in the research of digital twins and digital humans that can connect the virtual and physical worlds through interaction with various platforms, including social and economic activities. However, experiments and applications in the field of digital twins that connect the metaverse and humans are limited. In particular, research on expressing human emotions in the metaverse is also limited. Furthermore, this technology has the potential to be applied in various industries, including gaming, education, healthcare, and marketing, to provide new forms of interaction and experience. Therefore, in this paper, we propose a metaverse emotion mapping system that can inject new facial expression recognition that can be represented in metaverse objects. We implemented facial expression recognition using a deep learning model based on facial expressions captured by a webcam and applied emotion filtering. The filtered emotion data was then transmitted through an IoT server and the Metaverse Inter-working Module to build a system where objects in the metaverse change. This enables users to share emotions with others through metaverse objects, which could enable emotional communication and expression between people in the metaverse.