MotionControlMapper: A Gesture-based Real-Time Framework for Robotic Arm Coordination and Precision Control

Human-robot interaction (HRI) is becoming more prominent today in areas where robotic manipulators are controlled in a convenient way. This study describes MotionControlMapper, a new system for controlling a robotic arm through gestures in real time. This model relies on multi-data source processing and a pairing of Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM) and temporal attention systems to recognize human gestures and provide dynamic commands to a robot’s arm. Data from gestures is picked up by RGB-D cameras and inertial measurement units (IMUs) in real time and later preprocessed using Kalman filtering and time synchronization for high accuracy. Testing was done by using ten gestures from the robotic control dataset in various settings such as different lighting conditions. The accuracy of gesture recognition in this framework was 96.2% and the F1-score was 95.8%, much higher than those given by CNN-LSTM (91.2%) and Transformer (93.1%). It took robotic execution 22 ms less than the baseline systems, on average, for every task. Multitasking in messy or dim conditions became over 15% more successful than before. Also, people using the system reported that it feels more responsive and easier to use compared to before.