Media files to ISL: GAN based Indian Sign Language Interpreter

Indian Sign Language (ISL) is used as the mode of communication among deaf people in India. Currently videos, audio files, and any other synchronized or time-based media are not accessible to deaf or hard of hearing. The proposed Generative adversarial network (GAN) based system will automatically convert text or transcriptions from audio/video file to sign language in avatar form will aid them to understand the content in the media files. It is achieved by breaking down the conversion into two sub-processes. Text is converted to gloss sequences using Neural Machine Translation (NMT), and then it is converted to ISL pose sequences using GAN followed by generating corresponding avatar. Experimental results showed that the proposed sign language interpretation provides richer and more equivalent access to synchronized media