SLatAR - A Sign Language Translating Augmented Reality Application

Recent studies about Augmented Reality (AR) based interaction focus on speech and gesture, but it limits over 1.5 billion Deaf and Hard-of-Hearing (DHH) users from interaction in the context of remote collaboration. They primarily use sign language and facial expression as their main communication method, which is usually omitted in AR communication. Earlier works conducted several attempts to complement the limitation of DHH users’ interaction. By using speech recognition technology, researchers visualized the speech as a text in the AR space. Another approach augmented a translated sign language animation following images or videos. But these are only limited to text translation and are spoken-user-centred. There is a lack of studies that translate from speech to sign language in remote AR collaboration. To overcome this, SLatAR focuses on converting the speech spoken by the normal cognitive user to sign language using AR. This helps in building the communication gap between them. The speech generated by the user with normal cognitive ability is recognized and is converted into the Sign language gloss after undergoing various processes. A total of 51 distinct gloss words have been animated in this system which are mapped to its animation file and mimicked by the 3D avatar.