Video-Based Sign Language Translation System Using Machine Learning

The development of an interactive real-time video-based sign language translation system powered by efficient machine learning algorithms which is commonly developed for deaf-dumb people who are not able to hear or speak and is difficult for them to communicate among themselves or with normal people. Gesture and human activity recognition both are crucial for detecting the sign language as well as the behavior of an individual. These components are rapidly growing domains, enabling higher automation in households as well as in industries. Since extracting the features from continuous hand movements is complex and traditional sign language recognition gloves are costly, the combination of two deep learning algorithms, CNN and RNN can be used for automated sign language recognition. When both of these algorithms are used accuracy of the system also increases (the estimated accuracy is noted to be 92.4% on dynamic hand gestures studied on most of the available datasets). The system will then be able to translate the recognized sign language to desired text and then to speech for further communication using open-source Text-To-Speech API with python. This type of system has the potential in the future to enable a person to give the presentation or join a video conference in business or educational platform in which image or video-based representation of sign language can be projected as the person is speaking on a real-time basis. This architecture is well constructed and hence can solve many difficulties in communication purposes.

I. Introduction

Sign Language Communication is the most important and focused method of communication for deaf & dumb people. According to World Health Organization (WHO) report almost 466 million people worldwide have impaired hearing loss and studies expect these numbers might increase in the future [1]. It is very essential to develop a system that will make communication for deaf-dumb people much easier and will help to reduce the difficulties faced during the communication which exists between people who have a hearing impairment. Sign Language allows deaf-dumb people to communicate with the rest of the world. Sign Language involves more than 135 different sign languages out of which Indian, American, and British Sign Language (ISL, ASL, and BSL respectively) are widely renowned [2]. Most people rely on sign language interpreters for communication purposes. Though when it comes to finding a professional sign language interpreter it becomes unaffordable and difficult. Various other Sign Language Recognition and Translation Systems have been introduced recently are either costly and not that efficient. Hence our system is designed on a software platform that is much cheaper and will provide higher efficiency.