IoT-Oriented Gesture Automation with Mesh Detection Through OpenCV and Pyfirmata Protocol Using ResNet-Mediapipe

In this paper, we introduce a hybrid gesture automation model and an IoT control system tailored for practical use, enabling the conversion of hand movements into electrical signals for home appliance control. While devising our workflow, we enhance AI gesture-based automation with mesh detection through OpenCV for object detection and recognition on python and pyfirmata protocol. We seamlessly integrate it with the Pyfirmata protocol and Mediapipe to boost recognition speed and precision. Besides, we formulated an algorithm trained to access the loads with precise data from hand gestures. Our experiments validate that ResNet-Mediapipe effectively captures hand positions in diverse scenarios, even within confined spaces. Operating within a pre-trained environment, it attains real-time gesture recognition with exceptional accuracy, exceeding 98.5%, and a remarkable recognition speed of up to 45 fps, outperforming alternative gesture recognition. Our novel gesture-controlled IoT system is outfitted with essential hardware components, such as the AVR Atmega 328p Microcontroller, NodeMCU ESP8266 wifi-based module Microcontroller, 5v relay switch, and a range of IoT devices, all of which play pivotal roles in its operation. Our system not only achieves gesture recognition powered by ResNet-Mediapipe but alsoeffectively manages a multitude of IoT devices. It boasts the benefits of low power consumption, uncomplicated hardware needs, intuitive hand gesture control, and seamless integration into human-computer interaction scenarios with minimal delay.