First-person view hand posture estimation and fingerspelling recognition using HoloLens

Sign language users tend to be socially restricted due to the general population’s lack of knowledge of sign language. Some attempts have been made to develop technologies that improve this aspect by translating sign language. However; these approaches generally use a third-person camera for collecting the information, limiting sign users to environments prepared for this purpose.In this study, we develop a first-person view Japanese fingerspelling recognition system using an Optical See-Through Head Mount Display (OSTHMD). The system estimates the hand posture from the camera mounted on the OSTHMD and applies machine learning to the hand posture data to classify the hand gestures and convert them into speech. 37 Japanese sign language fingerspelling were successfully recognized by using a Microsoft pose extractor. Next, using a support vector machine, 37 out of 53 Japanese sign language fingerspelling were successfully identified with more than 70% identification rate. Finally, the specified labels were converted into speech using the speech output module with Azure API.The main purpose of this research is to propose a system that enables sign language users to communicate with verbal people without environmental restrictions.