3D hand pose estimation and action recognition in acupuncture with a representation learning framework

In this paper we propose a representation learning framework that combines semi supervised learning theory algorithms with domain adaptive methods. The model is trained with images of acupuncture manipulation for self-supervised representation learning to estimate 3D hand posture. The model demonstrates an End-point-error (EPE) of 0.39 cm and achieved a percentage of correct keypoints (PCK) score of $94.1 \%$ on the testing dataset. Through the analysis of the joint point synergy movement trajectories and patterns in acupuncture manipulations, the categories of lifting-insertion (LT) manipulations and twisting (TR) manipulations were successfully distinguished. Critical stimulation parameters such as LT amplitude, TR angle, and stimulation frequency were meticulously extracted. The representation learning framework provides an important tool for evaluating the stimulation parameters of acupuncture manipulations.