Real-Time Word Level Sign Language Recognition Using YOLOv4

Sign Language is a communication language for hearing and speech impaired people. To establish communication between hearing/speech impaired people and normal people, there is a need for an efficacious translation system. This paper proposed a real-time, vision-based sign language recognition system for translation from sign language to text based on deep learning. To achieve real-time recognition and translation, The proposed methodology uses You Only Look once version 4 (YOLOv4), a single stage detector to achieve real-time recognition and translation. For experimentation Indian Sign Language Dataset for Continuous Sign Language Translation and Recognition Dataset (ISL-CSLTR) is used. This dataset is extended further by adding images from additional two signers to increase the already available dataset. Pre-trained network is employed and fine-tuned on the extended ISL-CSLTR dataset. Data augmentation is also employed to increase the accuracy of the system. This proposed method achieves a mean average precision (mAP) of 98.4% and real-time translation. The results obtained makes the system applicable for real world applications which is the main aim of this experiment.