Evaluating the Accuracy of Real-Time Japanese Sign Language Word Recognition with Vision Transformer Models Trained on Angular Features

In recent years, deep learning technology has greatly advanced research designed to improve communication with hearing-impaired individuals. This research has particularly focused on improving systems for automatically recognizing and translating sign language. Our study introduces a new method that utilizes a Vision Transformer (ViT) to recognize Japanese Sign Language in real-time. This technique employs the Mediapipe pose estimation library to pinpoint the exact positions of finger joints within video frames and converts these positions into one-dimensional angular features. These features are then organized sequentially to create a two-dimensional input vector for the ViT model. We explored different configurations by varying the number of encoder layers in the ViT model to find the most accurate setup. The proposed method was further tested on video data representing words to assess its effectiveness, covering signs with and without actions. Our experiments with Japanese sign language videos showed the proposed approach's real-time capabilities and accuracy evalutation.