Real Time Conversion of American Sign Language to text with Emotion using Machine Learning

Auditory impairment or hearing loss is a major problem in today’s human population. Sign language helps hearing-impaired people to live their social life without much difficulty. The latest technology used in detecting sign language connects them to the rest of the world. Sign language recognition and conversion to the text based on the movement of hands and the shape formed by the fingers is a complex system. The solutions using machine learning give significant success for this complex system. This paper mainly focuses on developing a system for recognizing the different hand signs in American Sign Language and their emotions simultaneously in real-time and converting them into text. The system resolves the need for a translator to bridge the gap between a sign language based user and a non-sign language-based user. Most state-of-the-art technology involves CNN models with image pixels by identifying specific key point coordinates on the face/hand obtained. Using the latest technologies like MediaPipe, an improved CNN model is developed based on the distances between each unique identified vital point. The customized model able to achieve 80 percentage of accuracy for the live image.