Contactless Sensing for Recognizing Common Signs in ASL and BSL

Deaf-mute people use sign language to communicate using hand movements, body postures and facial emotions. Recognising automatic sign language is still a challenge and is in its early stage of research. For this, sensor-based and vision-based approaches are used; however, they have some limitations (privacy concerns, maintenance and ambient lighting). So, contactless sensing emerged as a promising solution to recognise automatic sign language. This work used contactless sensing to recognise ten common signs(I/me, You, Meet, Hello, Understand, Help, Sorry, Time, Candies/Sweets, and Thank you) used in american sign language and british sign language to make the system more efficient. To the best of our knowledge, recognising two sign languages is a novel work which is not done before. Raw data of a single participant is gathered, which is then converted into a spectrogram using signal processing. Afterward, six pre-trained deep learning models (EfficientNet, InceptionV3, MobileNet, ResNet5O, VGG16, and VGG19) are used for classification purposes. 92% accuracy of classification is achieved using ResNet5O and VGG19.