Enhanced Sign Language Translation Between American Sign Language and Indian Sign Language Using LLMs

This research introduces a foundational framework aimed at bridging the communication gap between American Sign Language (ASL) and Indian Sign Language (ISL) by translating alphabet-level gestures. The proposed system employs a hybrid deep learning model for ASL gesture recognition, integrating a random forest classifier (RFC) and a convolutional neural network (CNN) to enhance accuracy. Recognised gestures are converted into text, which is refined using a prompt-configured large language model (LLM) for contextual and grammatical accuracy. The corrected text is then synthesised into ISL gestures using RIFE-Net, a real-time intermediate flow estimation network, to generate smooth and natural gesture videos. The framework addresses key challenges such as gesture variability and linguistic differences between ASL and ISL. The hybrid model achieves a gesture recognition accuracy of 93.0%, measuring how accurately the system identifies ASL signs. Following recognition, the raw text output is refined using the Large Language Model (LLM), resulting in a text correction accuracy of 94.2%, which reflects improvements in grammatical correctness and contextual relevance. These metrics collectively demonstrate the system’s effectiveness in alphabet-level recognition and gesture synthesis, laying the groundwork for more advanced sentence-level translation. Initial experimental results demonstrate real-time processing capabilities, averaging one gesture per second, with video outputs at 60 FPS. This system not only facilitates seamless communication between ASL and ISL users but also lays the groundwork for scalability to other sign language pairs. The results highlight the potential to improve accessibility and inclusion of the global hard of hearing community, paving the way for future research in multi-modal sign language translation systems.