On-device hand model for robust gesture detection

The paper presents an on-device neural network for multimodal hand understanding. This allows to prepare robust gesture understanding using small additional datasets and low resources for gesture model preparation. It extracts 21 2D hand keypoints, a segmentation mask, a full kinematic hand model and verifies if the hand is in a frame and objects in hand. The proposed model can be deployed on various devices equipped with RGB or other cameras (such as PCs, TVs and mobile devices) due to its lightweight nature. An evaluation of the approach demonstrates high gesture recognition accuracy of 98.4% for real time on device solution.