Continuous Sign Language Recognition Through Transformers and MediaPipe Landmarks

Sign language is the primary means of communication for deaf individuals or those with hearing impairments. However, as the population does not widely use it, less development has been carried out, such as in verbal language translation. This work presents a recognition method for translating sign language into verbal language in real-time, aiming for accessibility comparable to that of modern verbal language translation. The proposed system utilizes MediaPipe to extract landmarks from video frames, employs a modified Transformer model to convert these landmarks into glosses, and fine-tuned a GPT-4o mini model to transform the glosses into verbal language. With the proposed methodology two models were trained. The first model was trained with available data from the SIGNUM dataset, where 92.93% of accuracy was achieved, with a signer that the train never visualized. The second model was trained with a database augmented with new sentences, concatenating isolated signs taken from the original. This model is used for real-time continuous translation and obtained an accuracy of 84.14%.