Hand Tracking in 3D Space using MediaPipe and PnP Method for Intuitive Control of Virtual Globe

Thereis a need for new interactive techniques to engage students with academic content. One of the hardest concept students deal with is using the Atlas and maps to understand the Geography, Social Sciences, and History of the world. The current alternative to the Atlas is accessing virtual globes on computers but controlling the globe is done using a mouse or touchscreen. The interaction may turn monotonous with usage. To address this problem, the authors propose a methodology to track a hand using a camera feed of a computer, while further using hand gestures to manipulate a virtual globe. Google's MediaPipe library is employed, along with OpenCV's pose estimation functions that use PnP (Perspective-n-Point) method. Later, an interactive globe environment has been implemented on a web browser using Cesium which can be commanded to perform rotate, pan and zoom on virtual globe and 2D maps as per the user's hand position and gestures. Results regarding the accuracy of the hand tracking system and robustness of control of virtual globe are also reported here.