Speech To Indian Sign Language (ISL) Translation System

The deaf and dumb population across the Indian subcontinent predominantly communicate with each other and with other people using the Indian Sign Language (ISL). ISL employs signs consisting of hand gestures, facial expressions or body postures to convey the desired message and emotion. It is a full-fledged natural language with its own grammar and lexicon. In order to lessen this significant communication gap between the hearing and speaking impaired community and the normal population, there is a need for translation systems. We propose an end-to-end human interface framework that is capable of recognizing and interpreting spoken language and then act out the corresponding ISL gestures to facilitate a very convenient, real time form of conversation between the disabled community and the rest of the population. We made use of the Microsoft Xbox Kinect 360s depth sensing and motion capturing abilities to capture motion data for all the different ISL gestures and then used Unity3D to set up all the animations and then finally bundle everything into an Android application.

I. Introduction

According to the 2011 census, 63 million people in India suffer from a from of significant hearing disability which adds up to 6.3% of the entire population of the country [2]. Of these 63 million people, it is estimated that at least 76-89% of them have no knowledge of Indian Sign Language [2]. The reasons for this comparatively low literacy rate are believed to be: 1) Lack of ISL interpreters 2) Unavailability of ISL tool 3) Lack of research on ISL. Speech and hearing impaired people face a lot of communication issues due to their limited ability. This poses quite a problem for the growth and development of disabled people. The most pervasive solution to this problem is the use of sign language for the purpose of communication. While the usual spoken languages are used by the hearing population to express thoughts and ideas, a sign language like ISL is a visual language based on gestures and action serves as the primary means of communication for hard hearing individuals.