Deep Learning Methods for Indian Sign Language Recognition

Indian Sign Language is the language used by specially abled people in India. Unfortunately the general population has no understanding of the sign language which hampers the communication between the specially abled and the general population. We are proposing a methodology to bridge this gap. We have used two approaches to solve this problem. First using the depth+RGB data captured using a Microsoft Kinect and predicting the gestures in real time. For segmenting the hand region from the data obtained by the RGB-D camera we used 3D reconstruction and affine transformation to map the depth and RGB information. Convolutional neural networks were used and segmented hand images/videos were used as an input to them. 36 static hand gestures from Indian Sign Language were trained and a classification accuracy of 98.81% was achieved on the test data. This model also showed a good performance when we transfer learned the American Sign Language giving a classification accuracy of 97.71%. LSTM with a convolutional kernel was used for training 10 dynamic gestures. This model achieved a classification accuracy of 99.08%. But as soon as we implemented this system, we figured out there is an inherent problem with this methodology. It is practically unreasonable to carry the bulky Microsoft Kinect around along with a system capable of performing the computation to communicate with people. We attempted to solve this problem using semantic segmentation of the hands. We used U-Net with ResNet 101 as the backbone for the same. Semantic segmentation utilises the input from a normal RGB camera which completely removes the necessity of using a RGB-D Kinect camera. We performed multi-class semantic segmentation which gave an IOU score of 0.9920 and an F1 score of 0.9957 on the training data. The above models performed extremely well in real time.