Human Pose-Estimation and low-cost Interpolation for Text to Indian Sign Language

Throughout India, the deaf community comprising millions of people uses Indian Sign Language. However, India houses only about 300 certified interpreters causing a lack of trained teachers and teaching materials in the country. This creates a language barrier with no means for translation without an interpreter. Previous approaches for creating a translation medium between spoken languages and sign languages are heavily dependent on Motion Capture technologies which are extremely expensive and not ideal for use by non-professionals even though highly accurate. Thus, computer vision approaches can be adopted, however, such approaches are highly dependent on huge labeled datasets. Datasets for sign language are hard to acquire and even more so for Indian Sign Language which has received little to no predominant attention in the past years. Based on that, this paper aims at generating 2D pose data of Indian signers from existing single-camera RGB videos. However, the videos are expected to be of varying quality, made using differently positioned cameras and consists of multiple signers against unified single-signer datasets. Unlike other applications, such as gait analysis, activity recognition, for sign-language face and hand key points play a major role. This paper represents the sub-project which is planned to be implemented as the preliminary stage to an actual ISL translation medium. In this paper, video is initially stabilized to reduce jitters and correct the perspective errors in videos. Then, 2D human-pose key points from the video are extracted and saved. Further, this saved data is analyzed and solved for missing key points which arise due to occlusions inherent in signs. For this, a low-cost trajectory interpolation method is used to fill the missing joint points. Overall, this vision-based motion capture approach gives a fairly non-distorted 2D pose model that is further corrected by using the information from neighboring frames simultaneously reducing time and computational cost. The resulting dataset serves as the base for further phases and research work.