Gesture Based Virtual Assistant For Deaf-Mutes Using Deep Learning Approach

Roughly 5% of people around the world are deaf-mutes which imply they face difficulties in interacting with normal people, sign language is communication media for them. Gestures made through hands are one of the nonverbal communication modalities used in sign language. In order to communicate among themselves, the speech and hearing impaired utilize sign language as they have an understanding of conventional sign language. However, they are unable to interact with the outside world because most people are not aware of the presence and use of sign language. This technology tries to break down the hurdle between the disabled and everybody else in the world by identifying and converting hand signals into text in English and Kannada languages which are further translated into speech. Convolutional Neural Networks, Tensor flow, Deep Learning, Python text-to-speech module pyttsx3 and OpenCV are among the concepts used. A webcam captures the hand motions and is processed using the OpenCV python library, which is eventually interpreted by the deep learning algorithm called Convolutional Neural Network which generates rational languages. To map these rational languages to their pre-defined datasets we make use of deep learning, and to satisfy this requirement neural networks are used in conjunction with the Tensor Flow Library. Once the gesture is recognized it will be displayed on the viewing screen and instantly converted to speech. This system acts as a two-way virtual assistant that helps in converting sign-to-text and speech and thus assists the disabled in the utilization of modern technologies for communication.