Early Grasp Prediction With Incomplete Data via Spatial Gating and Temporal Weighting for Teleoperation

Accurate and prompt speed grasp intention recognition is crucial in online human-robot interaction (HRI). However, dynamic grasping relying on complete motion for high recognition accuracy will lead to an unavoidable delay in real-time prediction. To address this issue, we propose a Spatial Gating and Temporal Weighting Early Grasp Prediction (STEGP) method that utilizes incomplete dynamic grasping data from sliding windows to reduce the time delay for reliable robot teleoperation. The proposed method comprises a synergy-based feature extraction module, a spatial gating classification module, and a time-decay weighting fusion prediction module. The spatial-temporal mechanism with gating units effectively classifies sequential movements, achieving performance comparable to that of Transformers but being much easier to implement. Integrating a time-decay weighting frame enables reliable early prediction even with incomplete data. gMLP is chosen for the classification of hand dynamic grasping because of its high accuracy, realizing 93.83% accuracy for 33 grasping categories. The prediction tests demonstrated 85.4% accuracy, with the first 25% grasp completion across 28 subjects. Online robotic teleoperation grasp experiments achieved a 57.4% reduction in time delay and a 93.3% success rate. Note to Practitioners—Gesture recognition-based robotic hand control offers the advantages of intuitive operation and incremental configuration. However, motion/gesture recognition can cause delays in the movement of the robotic hand due to the lag associated with the operation of hand motion and the recognition time of the model. We propose a grasping prediction framework that leverages incomplete hand movements for recognition to address this issue. We achieved high recognition accuracy at the early stages of hand motion for 33 types of grasps. Experimental results demonstrate that the delay of robotic hand execution relative to the human hand is significantly minimized. The study enhances the efficient control of the robotic hand through the natural interaction of the human hand grasping.