Empowering Physical Attacks With Jacobian Matrix Regularization Against ViT-Based Detectors in UAV Remote Sensing Images

Vision transformers (ViTs) have achieved great success in unmanned aerial vehicle (UAV) target detection tasks. However, little attention has been paid to the adversarial attack against ViT-based detectors, and the generated adversarial examples cannot take physical realizability and attack transferability into account at the same time. To overcome the limitation, we focus on transferable attacks toward ViT-based detectors in optical UAV-based remote sensing images and generate adversarial examples in the physical world. Concretely, we design unique perturbation patches deployed within and beyond the target object rather than requiring the patches to be aligned with image tokens. To narrow the gap between limited digital samples and complex physical scenarios, we conduct data augmentation on training images at global and local levels. In addition, we propose a novel transferable attack method named Jacobian matrix regularization (JMR), which consists of feature variance regularization (FVR) and attention weight regularization (AWR). Specifically, FVR calculates feature variances of different channels within specific layers and then sets the features as zeros for channels with top variances. AWR is achieved by masking the largest self-attention weights. We conduct extensive transferable experiments with typical detectors in both digital and physical UAV-based remote sensing scenarios. The results indicate that our method could achieve competitive transferability compared with state-of-the-art methods.