Automated Indian Sign Language Recognition System Using LSTM models

According to a survey by the World Federation of the Deaf, there are approximately 1.5 billion deaf people around the world, and more than 80 percent of them live in developing countries. In India, around 63 million people suffer from complete or partial deafness, and of these, at least 50 lakh are children. There is a separate community of people suffering from deafness because of the inability to communicate with the common people. Also, the common mass of the majority of developing nations isn’t aware of the different ways they can communicate with people suffering from deafness or dumbness due to which a communication gap is formed, which restricts deaf and dumb people’s ability to fit into society and contribute to the development of society to the best of their ability. To overcome this, many new sign language recognition models have been introduced. Hence, this inspired us to create a system that is especially for Indian Sign Language (ISL) users known as the Indian Sign Language Recognition System (ISLR). But the major challenge was the way to recognize the signs as Signs of Indian Sign language are a bit complex, it consists of signs involving both the hands of the person thus making it hard for an existing CNN based model to detect the signs. To overcome this mediapipe holistics landmarks are used, which gives value specifying the position of different landmarks of both the hands. For different signs these values will be different and the LSTM model is the best choice to work on these types of data. So a system working on Mediapipe and LSTM is created. Three LSTM models: simple LSTM, bidirectional LSTM models, and stacked LSTM models are being used in this system. A synthetic dataset consisting of seven gestures and 26 alphabets was created to train the model. The gestures taken are-Hello, Thanks, I love you, Thumbs up, Home, Help, Namaste. A dataset, inclusive of 30 videos of 30 frames each of certain gestures was recorded to train the model. Different performance measurement parameters like Accuracy, Precision, Recall, F1 Score, and Cohen Kappa Score are used in this work to compare the three LSTM models. As result, we have observed that simple LSTM and bidirectional LSTM models outperform stacked LSTM models.