Gesture Prognosis: Interactive Sign Language Learning using Real-Time Gesture Recognition

This research study proposes and implements a Sign-to-Text translation system to bridge communication gaps for the Deaf and Hard of Hearing (DHH) community. The proposed system must utilize improvements in computer vision, natural language processing, and machine learning to interpret gestures in American Sign Language and translate them into audible text in real-time. The proposed system employs CNN for hand and body gesture recognition and RNN for sequence prediction for accuracy in fluency, that is, in translation. Preliminary test runs seem to be very promising and the system was able to recognize 92% of common ASL phrases. This research study discusses about the proposed architecture, training methodologies, and performance evaluations of the Sign-to-Text system; it points towards its potential to be used ultimately in enhancing accessibility and inclusiveness for DHH individuals across a range of social and professional environment.