Sentence Level Indonesian Sign Language Recognition Using 3D Convolutional Neural Network and Bidirectional Recurrent Neural Network

Sign Language Recognition (SLR) is a relatively challenging research field which allows opportunity for improvements. In this research, we propose sentence-level SLR using deep learning method by combining Convolutional Neural Network (CNN) and Bidirectional Recurrent Neural Network (Bi-RNN). Specifically, 3D CNN is implemented to extract features from each video frame and bidirectional-RNN is implemented to extract the unique features from the video frame's sequential behavior, which later generate a possible sentence. There are two key takeaways from this paper. The first is our proposed dataset of Indonesian Sign Language (SIBI) which is comprised of 30 sentences in SIBI. The second is our novel approach of using deep learning and Connectionist Temporal Classification (CTC) loss function in sentence-level SLR. The result shows that the hyperparameter used, in this case Hyperparameter 1, achieves the best result. Also, this research found that deeper network does not necessarily guarantee good results. A bigger number of dataset also affects the performance of the system.