Hand Gesture Recognition System with Finite State Machine for Remote Desktop Control

Humans may interact with computers in many ways. One of them can be communication through recognition and right interpretation of hand gestures. This study addresses the problem of remote desktop control through visual-based human–computer interaction that is based on hand gesture recognition. We designed and implemented the hand gesture recognition system that combines various machine learning (ML)-based technologies, including the MediaPipe Hands model for detection of keypoints on hands, and the artificial neural network for classification of single static gestures. To recognize dynamic or group gestures that are assigned to command actions, we proposed a flexible recognition system that is based on the concept of finite state machines. The new group gestures do not require retraining and can be easily composed of single existing gestures, making our approach more accessible to the end user. The experiments demonstrated that our system can correctly recognize a few group gestures for performing the following operations on the computer desktop: cursor movement, clicking, window dragging, full-screen switch on/off, zooming in/out, window closing, and active window switching.