Indian Sign Language Recognition Using Video Vision Transformer

Sign language is a way of communication that uses hand shapes, orientation, movements, and facial expressions to express instead of spoken words like normal language. Different regions have developed their own versions of sign language. In India, Indian sign language (ISL) is the primary mode of communication for more than 5 million speech or hearing impaired people. However, the knowledge of signing and understanding the ISL is only limited to the deaf community, which can make it hard to communicate with the people who don’t understand the language. Thus, effective recognition of ISL gestures is vital for seamless interaction between these impaired people and the broader society. The rise of artificial intelligence and computer vision models has provided us with many effective tools to tackle this challenge. While there has been some progress in this field, most research has focused on static signs or American Sign Language, leaving video-based recognition for ISL largely unexplored. To bridge this gap, our paper proposes an approach that utilizes Video Vision Transformer (ViViT) architecture for ISL recognition. We have experimented with different layers and different numbers of attention heads to get a model that works best for the assigned task. Our architecture gives an overall accuracy of $\mathbf{9 6. 6 9 \%}$ and a top-5 accuracy of $\mathbf{9 9. 5 5 \%}$ on the sign recognition task using our own created VISL-PICT dataset.