Virtual Hand-Gesture Interaction System based on SRHandNet with High Frame Rate

Hand gesture recognition plays an important role in virtual human-computer interaction systems. However, existing methods often rely on a variety of smart sensors or data gloves, which significantly increases the cost and threshold of research in this area. Therefore, this paper sets out to hand key points recognition from a low-cost monocular camera. Previous research on hand key points recognition with monocular cameras suffer from high latency and instability since they process each frame independently via a traditional deep learning backbone that cost much computing power. To alleviate this issue, we build a real-time interaction system with a monocular camera based on SRHandNet, which is a simplified and qualified deep neural network model on hand detection. We modify the original model with TensorRT and Kalman Filtering, intending to achieve a more stable and efficient application for both windows and edge platforms(e.g. NVIDIA Jetson), on which the capturing frame rate of the interaction system is doubled. With the estimation of camera position, we realized the mapping between 2D and 3D hand key points, achieving virtual human-machine interaction. A series of experiments validated the superiority of the proposed model over baselines, suggesting that this work may have great prospect in model deployment and hand pose recognition in augmented reality applications.