Sign2Text: Deep Learning-based Sign Language Translation System Using Vision Transformers and PHI-1.5B

Sign language is essential for communication among deaf individuals, yet barriers persist effectively in translating its rich linguistic expressions into textual representations. The dynamic nature of signing poses a significant challenge for existing sign language recognition systems, particularly in intricate and continuous contexts. Moreover, these systems primarily cater to specific sign language dialects, neglecting the diverse linguistic landscape of sign languages globally. This paper proposes an innovative sign language translation system that addresses these challenges by leveraging recent advancements in computer vision and natural language processing. Our approach focuses on Indian Sign Language (ISL), which presents unique challenges due to its utilization of both hands. The system utilizes a Vision Transformer (ViT) trained on a comprehensive video dataset to classify various sign language elements, while integrating a sophisticated language model, PHI-1.5B, to refine translated text for grammatical correctness and structural integrity. By combining ViT and PHI-1.5B, our system aims to achieve robust and contextually relevant translation of ISL gestures into textual representations.