Real-Time Sign Language Recognition using Deep Learning Techniques

Sign language uses hand gestures, shifts of hand shapes, changes in body postures, and even facial emotion to recognize the actions conveyed and trace that information meaningfully to express it conventionally. It is the prime communication medium for people with hearing disabilities and language impairment. We have developed an application that can detect the most popular greeting sentences in American Sign Language. Our application gives real-time predictions of these sentences and words with an average accuracy of 75% for every word. For Gesture identification, we used a dual-model system of Single Shot Multibox Detector (SSD) and Machine learning model working on 2D-Pose coordinates of the user at runtime. Our application also supports ASL Alphabet detection in real-time to generate custom phrases. This module leverages Google’s Inception v3 for transfer learning and gives an accuracy of 89.91%. All the Machine learning and Deep Learning models for our application were trained on a custom dataset created by us.