Mixed reality (MR) Enabled Proprio and Teleoperation of a Humanoid Robot for Paraplegic Patients

Paraplegia is a disability caused by impairment in motor or sensory functions of the lower limbs. Most paraplegic subjects use mechanical or motorised wheelchairs for their movement, however, this may limit the capability of patients to independently perform common activities of daily living (ADL). In this paper, a novel mixed reality (MR) enabled proprio and teleoperation framework for a humanoid robot is presented. The framework can be operated by a paraplegic person by using inputs from an MR headset. The framework enables varied and unscripted manipulation tasks in a realistic environment, combining navigation, perception, manipulation, and grasping. The impaired operator can make use of a wide range of interaction methods and tools, from direct teleoperation of the robot's full-body kinematics to performing grasping tasks or controlling the robot's mobile base. The adopted humanoid robot is the EVEr3 Humanoid Research Robot from Halodi Robotics, while the Oculus Rift S is chosen as MR headset. To demonstrate the potential of the proposed framework, a human subject study is presented. In this study, a home/workplace environment is rendered with MR by combining physical shelves and everyday objects, such as goods to be grasped, with simulated elements, such as the robot avatar and the control interface. A paraplegic subject is involved in the study. Results suggest that the proposed MR-enabled system improves the patient engagement and illusion of presence.