An IoT-Oriented Gesture Recognition System Based on ResNet-Mediapipe Hybrid Model

In this paper, we propose the ResNet-Mediapipe hybrid model and a new Internet of things (IoT) control system in order to implement gesture control of IoT devices. In the process of designing the ResNet-Mediapipe workflow, we improve the ResNet-50 and integrate it with Mediapipe to improve recognition speed and accuracy. Besides, we also create a brand-new gesture dataset with more than 5700 pictures, which is richer than other open-source gesture datasets. Experiments show that ResNet-Mediapipe can capture hand position in a variety of situations and recognize gestures in real-time with an accuracy rate of over 98.5% and recognition speed up to 45 fps, which is more accurate and faster than other gesture recognition methods. The new gesture-controlled IoT system we propose is equipped with hardware that includes Raspberry Pi, STM32F103, ESP8266, EG200U, and various IoT devices. This system can realize gesture recognition based on ResNet-Mediapipe and control numerous IoT devices. Besides, we design a human-computer interface for the system on the Ali IoT cloud platform and a set of gestures used for human-computer interaction, which enable the users to monitor IoT devices' information on the Web page and choose different IoT devices to control. The experiment proves that this project provides a precise and convenient way for people to interact with gestures.