American Sign Language Recognition using Video Vision Transformers

Sign language is the most natural and effective way for communication amongst the hearing/vocally challenged and the hearing abled. To bridge the social gap, this paper proposes a system that aids in translating from a video feed of American Sign Language signers to a known set of common English words. Current solutions for vision based sign language to English translation is limited to a very small set of target classes and old modeling techniques that do not take into consideration the temporal factor that caters to the sequential nature of signing. We use the WLASL dataset, which is a largescale dataset for Word-Level American Sign Language and it comprises of 2000 words from over 100 signers and develop models on WLASL100 and WLASL300 each representing the top 100 and 300 most commonly appearing glossaries respectively. For modelling, we make use of video vision transformers (ViViT) to leverage the power of multi head attention in a temporal and spatial setting and experiment with two feature extraction models like Truncated Resnet50 and MoveNet-thunder which act as a precursor to the ViViT. The final model does better than state of the art models like Pose-GRU, Pose-TGCN and VGG-GRU, which were also trained on the WLASL dataset with a top-1 accuracy of 59.44% on the WLASL100 subset and a top-1 accuracy of 48.38% on the WLASL300 subset.