Video-based Sign Language Recognition using Recurrent Neural Networks

Deaf-mute people regularly confront communication challenges with people who can hear and speak as the primary means of communication they use is sign language. People often do not have the time and knowledge required to interpret gestures of people relying on sign language and hence either the people feel neglected or rely on a professional interpreter. This could be problematic, however, especially in emergency situations like murder, road accident, robbery, etc. where an interpreter might not be immediately available and there is an urgent need to communicate. In this paper, we study sign language recognition by implementing deep learning techniques on the open-source Word-Level American Sign Language (WLASL) Dataset. We perform hand recognition on the video frames followed by feature extraction using state-of-the-art convolutional neural networks and then perform classification on the image sequence by using a recurrent neural network. We compare the performance of several architectures by using various metrics like validation and test accuracy, precision, recall and F1-score. We also test the working of the system on different sizes of label set to check possible scalability.