An Interaction Mechanism for Virtual Reality Based on Upper Limbs Motions Tracking Using Depth Cameras and Inertial Sensors

The evolution of the natural interaction between man and computer has represented a positive and promising impact for Virtual Reality (VR) applications. There has been a growing interest in developing new approaches and technologies to improve the user experience so that it can be as natural and immersive as possible. In this context, this work aims to introduce a new concept of natural interaction using the upper limbs with the combination of two types of sensors, classified here as Wearable Inertial Measurement Units (WIMUs) and Head-Mounted Depth Cameras (HMDCs). While HMDCs allow precise tracking of the forearm, hand and fingers, their limited field of view restricts the range of the movements. On the other hand, the WIMUs offer more freedom of movement, since they are not based on cameras and computer vision. However, they are not accurate enough to capture the limbs positions and in details the hands motions. Our solution presents a strategy to combine both classes of sensors in order to improve the user experience with a robust natural interface control. To test the solution, a VR game based on the use of the proposed strategy was developed. An study with tests and evaluation was also developed with users and the results show that the proposed solution outperforms the use of the sensors separately, mainly in terms of performance and fun. Although our proposal is focused on VR games, it can also be an important interaction interface for any other VR based application.