Interactive Communication Assistance System Based on In-Air Handwriting Recognition

The idea for this project was born out of the necessity to enhance communication for people dealing with either short-term challenges, such as those arising from surgery or illness, or long-term challenges, such as individuals with hearing or speech impairments. We've noticed that individuals with hearing impairments often resort to using sign language as a means of communication, but the responses they receive may involve gestures or partial interpretation of body language. Unfortunately, not everyone is skilled in sign language, which can lead to misunderstandings, misinterpretations, and difficulties in conveying precise information. This challenge becomes even more significant for individuals who temporarily cannot speak or hear due to medical conditions. Therefore, our goal is to resolve these concerns by creating a lightweight and user-friendly device. The proposed system utilizes Jetson Nano as the computational core, leveraging the Mediapipe toolkit to recognize the user's finger joints and gestures within the camera's field of view. It then employs OpenCV image processing and convolutional neural networks to recognize airborne handwriting. An OLED screen functions as the user interface for displaying restaurant information. The ultimate outcomes are showcased on this OLED screen and audibly conveyed through a speaker to facilitate communication with others.