Estimating Finger Joint Angles with Wearable System Based on Machine Learning Model Utilizing 3D Computer Graphics

Robotic rehabilitation for paralyzed hands utilizes exoskeletons and soft gloves equipped with active mechanisms to provide support for hand motion. From a safety and control perspective, it is imperative to measure finger joint angles during motion support provided by a soft robotic wearable system. However, embedding sensors into these devices can be inconvenient as it may lead to bulkiness or structural difficulties. This study aims to develop a machine learning model for estimating finger joint angles from images utilizing data created with computer graphics (CG), and to validate the feasibility of this method through basic experiments. The three-dimensional CG (3DCG) hand model includes bones corresponding to the major joints of the fingers, and a wearable system of the index finger imported from a computer-aided design software was attached to the 3DCG hand model. After rendering the integrated motion between the hand model and the wearable system for finger flexion and extension, images in conjunction with the finger joint angles were recorded as the training data. The finger joint angle estimator was based on a pre-trained vision transformer model and was learned using the created 3DCG training dataset. The basic experiments showed that the developed machine learning model enabled the estimation of finger joint angles with reproducibility for four types of hand postures with the wearable system. Furthermore, the differences between the joint angles measured in the real world and those estimated by the developed model were smaller than those for an existing hand pose estimation model. The developed machine learning model, which utilizes 3DCG, has the potential to estimate finger joint angles with the wearable system using image videos.