3D Hand Pose Estimation and Gesture Recognition Based on Hand-Object Interaction Information

Gesture recognition and hand pose estimation are two important tasks in the field of computer vision; however, in hand-object interaction scenarios, it is still challenging to implement gesture recognition and hand pose estimation using monocular RGB images since the hands are often obscured by objects. Most current studies treat these two problems as separate tasks. This paper proposes a multi-task network that combines gesture recognition and hand pose estimation tasks in a single network to solve this issue. The network uses a novel hand-object feature interaction module to improve the accuracy of gesture recognition and pose estimation by reasoning about the context between hand and object features through Transformer that encodes hand features and object features in different manners. In addition, to improve the performance of the multi-task network, we use a collaborative learning method that combines the output features of different tasks to promote each other, thus further improving the overall performance. Our model achieves high accuracy and performance for both gesture recognition and gesture pose estimation tasks on both the FPHA and HO-3D datasets, demonstrating the effectiveness and feasibility of the method.