Gloss-Free Sign Language Translation using Sign2gpt-Next Technique

This work proposes a new approach to convert the data from sign language to spoken language without exposing the data to a gloss layer. Whenever gloss annotations are used which frequently are incomplete and act as an intermediate representation, it is usually found to be a bottleneck in translating sign language videos. In order to avoid this, the authors use a combination of modern machine learning algorithms to arrive at the conclusion. They use Dino-V2 Vision Transformer to implement the function of extracting the spatial features of the input sign language video frames. This model is particularly suitable for learning and visual patterns in general and particularly for capturing fine details of these patterns. Further, a spatio-temporal transformer layer is added to automate the temporal processing of the Sign-Language which helps in the arrangement of the body gestures and limb movements needed at a specific temporal frame. It is then translated into the desired spoken language using a multilingual Generative Pre-trained Transformer (GPT) model. The approach can be said to be a big leap forward in sign language transliteration especially for such situations where few Gloss annotations are available, and making way for better and improved tools for the Deaf and the hard-of-hearing.