A Deep Learning Solution for Arabic Words Sign Language Recognition in the Context of Sentences

Sign language recognition is used to facilitate the communication between the hearing and the deaf societies. This paper proposes a video-based approach to recognizing Arabic sign words in the context of sentences. This is important as in real life the start and end positions of the arms are not necessarily at the sides of the signer. Rather, the position depends on the previous and successive words in the same sentence. This paper proposes to use video coding techniques to generate Sum of Displaced Differences Images (SDDI) of sign words using optical flow followed by the summation of the prediction errors of video frames. Time-dependent feature vectors are then generated from these images using transfer learning. Consequently, model generation is performed using recurrent neural networks. Experimental results performed on 80 sign language words represented in 40 sentences revealed that the proposed solution results in a word recognition rate of 98.6%, which surpasses the accuracy of existing work that uses the same sign language dataset.