A Multimodal Natural Interaction-Based Mixed Reality System for Limb Rehabilitation

The virtual rehabilitation system is designed to aid in the recovery of patients’ motor functions and cognitive disorders. It employs technologies such as virtual reality, human-computer interaction, and sensors. Compared to traditional rehabilitation systems, virtual rehabilitation systems have many advantages, such as providing multiple therapeutic scenarios, high entertainment, strong system replicability, and accurate quantitative evaluation. However, current virtual rehabilitation systems still face challenges, including a limited variety of training content, insufficient interaction experience, and a single mode of interaction, which negatively affect the practical experience of patients. To address these problems, this paper proposes a multimodal natural interaction technology based on gestures, tangible objects, and natural walking for patients with limb impairments, and designs a mixed reality rehabilitation system. User research showed that compared to the controller interaction mode, the multimodal natural interaction mode is more interesting and has a stronger sense of realism, enhancing participants’ enthusiasm and motivation.