Shortcut-Enhanced Multimodal Backdoor Attack in Vision-Guided Robot Grasping

Integrating the Artificial Intelligence (AI) vision module into the robot grasping system can significantly improve its generalizability, thereby enhancing the efficiency of Human-Robot Interaction (HRI). However, the inherent lack of interpretability in AI also opens the gate to external threats. In this work, we reveal a novel safety risk in this vision-guided robot grasping system by proposing the Shortcut-enhanced Multimodal Backdoor Attack (SEMBA), which can manipulate the grasp quality score using the backdoor trigger leading to a misguided grasping sequence. The SEMBA may thus cause potentially hazardous grasping and pose a threat to human safety in HRI. Specifically, we initially present the Multimodal Shortcut Searching Algorithm (MSSA) to find the pixel value that deviates the most from the mean and standard deviation of the multimodal dataset, along with the pivotal pixel position for individual images. This will guarantee that the proposed attack is effective in complex, multi-class object scenarios. Next, based on MSSA, we devise the Multimodal Trigger Generator (MTG) to create diverse multimodal backdoor triggers and integrate them into the dataset, ensuring that our attack has the multimodality attribute. We conduct extensive experiments on the benchmark datasets and a cobot, showing the effectiveness of the proposed method both in the digital and physical worlds. Our demo videos are available in supplementary items. Note to Practitioners—Robot grasping systems are typically designed to be safe and reliable in HRI scenarios. However, integrating an AI-powered vision module into such systems can introduce substantial unpredictability, particularly when relying on third-party data. This work introduces a novel backdoor attack method to reveal a new safety risk in this vision-guided robot grasping system. Our method aims to mislead the robot into performing hazardous grasps by altering the grasping sequence. It emphasizes the attack’s effectiveness in complex, multi-class object scenarios and highlights its multimodal nature. To the best of our knowledge, this is the first study to present backdoor attacks in vision-guided robot grasping. Our approach paves the way toward future AI-powered visual grasping safety studies and provides valuable insights into building a more reliable and trustworthy vision-guided system for cobots.