Real-time Sign Language Recognition Using Computer Vision and AI

Promoting inclusive communication is essential and sign language plays a crucial role in achieving this goal. In this research, a system is developed capable of making it easier for those who primarily communicate through sign language to interact with those who are unable or may not be knowledgeable in the language. This has the potential to help bridge the communication gap between those who are fluent in sign language, and those who may be struggling to learn or are not knowledgeable. In its current state, our system is capable of recognizing two forms of sign language, namely: American Sign Language and British Sign Language. The system also is capable of performing facial expression analysis to account for non-verbal inflections expressed by the user. These tasks are accomplished by making use of computer vision provided by the OpenCV Python library. It also uses various machine learning models and the MediaPipe library. We explore two approaches for sign language recognition: contour-based recognition and landmark-based recognition. Additionally, facial landmarks for facial expression analysis are investigated which can be used to detect expressions and inflections from a user's face alone. The next steps of this research will consist of working with more complex words and phrases and investigating gesture recognition.