Hand-Gesture Based Aerial Digital Canvas System Using Deep-Learning Algorithms

Human Computer Interaction (HCI) is an inter-disciplinary science that explores how humans interact with machines and other technologies in the everyday world. Hand gesture recognition, being one of the major applications, focuses on revolutionizing the way users interact with digital systems. The objective of our system is to provide an interface to users where they can write, draw, compute equations, use shapes, and also virtually type in the air, eliminating the use of physical tools. The existing systems used algorithms such as K-Means, RCNN, YOLOv3, etc. for detecting and tracking the hand movements. Although these algorithms were effective, they often slowed the system's performance to interact in dynamic environments. Inculcating the features from OpenCV and MediaPipe, our system can smoothly track hand movements as well as gestures converting them into digital strokes in the air. The system is also integrated with two CNN models to achieve features like solving Handwritten Mathematical Equations and robust shape suggestions with accuracies of 94.3 and 94.85 respectively. Using the virtual keyboard users can perform typing actions in the air, avoiding the use of a physical keyboard. The system's integration with these various features makes it highly compatible, user-friendly and unique from existing platforms which offer only one specific feature.