Dynamic Sign Language Recognition and Emotion Detection using MediaPipe and Deep Learning

Sign languages facilitate natural interaction by lowering the communication barrier between the hearing-impaired and society. Recognition of words in sign language has always been a difficult task if the gestures of two more words turn out to be similar. Moreover, due to the rapid switching of gestures while communication, creating sentences with the assistance of recognized words is a mammoth task. Further, recognition of alphabets and numbers will not help the user to convey sentences and emotions effectively. Therefore to address the outset issues, this paper proposes a dynamic sign language recognition and emotion detection (DSLRED) model that uses MediaPipe’s holistic pipeline along with an LSTM network which also classifies the emotion on a person’s face into one of seven categories, using deep convolutional neural networks. The results shows that the proposed model achieved an accuracy of 98.95% and can effectively convey emotions and sentences in sign language, independent of the signer who performed the actions in the training data.