Top-Down Pose Estimation Method Based Human-Computer Interaction for Smart Space System with Digital Twin

Human-Computer Interaction (HCI), a technology for human interaction with computers, has been studied a lot for a long time. As technologies related to the metaverse have recently developed, digital twin technology is also used in various industries. In the field of computer vision, various deep learning-based algorithms such as object classification, object detection, and pose estimation have been developed. In this paper, Using a deep learning-based top-down pose estimation algorithm, keypoints are extracted from three images and matched in a 3D virtual environment to create a digital twin. The coordinated digital twin delivers information to IoT devices in the real environment through actions that cannot be simulated in the real environment, such as shooting lasers in a virtual space. Customized actions such as opening doors and turning off lights can be performed through IoT sensors and actuators in real environments. The experiments are performed using Unity, and the results showed 82.67% accuracy on average.