An End-to-End Vision-Based Approach for Indian Sign Language Recognition and Captioning

Individuals who are deaf or hard of hearing (DHHr), particularly having 90% or more hearing loss, primarily use sign language as their mode of communication. The number of DHHr individuals worldwide is substantial and continues to grow, with India playing a significant role in the demographic. The study presents a vision-based application that facilitates communication between signers and non-signers by translating Indian Sign Language (ISL) gestures into English text. Recent advancements in deep learning and computer vision have shown promise in recognizing motion and gestures in sign language. The proposed model processes sequences of video frames to extract meaningful features. Specifically, frames are extracted from ISL videos to capture facial expressions and hand gestures. Using the iSign dataset, key point values for hand gestures and facial expressions are obtained from video frames. The extracted features are then processed using text tokenization and encoding techniques, associating gesture key points with their corresponding sentences in ISL. A Temporal Long Short-Term Memory (LSTM) model is trained on these features to generate sentences based on ISL gestures, which are then translated into English text. The approach significantly reduces the communication barrier between signers and non-signers. The proposed framework achieves 98.75% accuracy on the iSign ISL dataset, demonstrating its effectiveness in real-time ISL-to-text translation