Real-Time Bangla Sign Language Detection with Sentence and Speech Generation

Sign language, the non-verbal language used by the people with hearing and speaking disability, known as the deaf and mute, to connect the bridge of communication with others. Being a visual means of communication, it deprives the mutes to communicate with people having a visual impairment. A medium recognizing sign language and converting it into text and speech could fill the gap. Recently lots of research has been on Bangla Sign Language(BdSL) classification, not on Bangla sentence and speech generation. Most of them classify either digits or alphabets and also face time delay. This paper proposes a system for BdSL recognition that can interpret BdSL from a sequence of images or a video stream and generate both textual sentences and speech in real-time. We have used YOLOv4 as the object detection model. We have also proposed three new signs for the sentence generation task and built a dataset consisting of 12.5k BdSL images of 49 different classes where 39 are Bangla alphabets, 10 are Bangla digits, and the three new proposed signs.