Marker-enhanced Hand Tracking with Deep Learning

To meet the accuracy requirements of hand motion capture in virtual simulation environments for large-scale complex product evaluation, we present a finger-level hand tracking method by arranging the markers on the hand key joints and using multiple RGB cameras as the motion capture devices. Our method can initialize marker labels whenever needed by using CNN to predict hand joint positions as prior knowledge and labelling each marker based on the prior knowledge. Since the regular training datasets mainly focus on training neural network for capturing bare hand motion, we propose an image restoration algorithm allowing to continue using existing datasets to train CNN models and make sure the high accuracy of hand joint prediction in application scenarios. Verification experiments show that the root mean square error of our method is 3.9mm, which can meet the accuracy requirement in large complex product virtual simulation. Meanwhile, our method has good robustness for hand self-occlusion.