Inheritance Attention Matrix-Based Universal Adversarial Perturbations on Vision Transformers

Universal Adversarial Perturbations (UAPs), which is type of image-agnostic adversarial attack, has been deeply investigated for Convolutional Neural Networks due to its high efficiency. On the other hand, as an architecture based on self-attention mechanism, Vision Transformers (ViTs) have boomed and been widely applied to solve various computer vision problems since most recent years. In this letter, we delve into the robustness of ViTs against universal adversarial attack and propose an inheritance attention matrix-based UAPs (IAM-UAP). Specifically, we introduce the inheritance attention weight matrix (IAM), which represents the integration of global information of the input patch sequence. Further, we propose a perturbation optimization objective based on IAM to confuse the global information integration of ViTs. The empirical results confirm the attacking capability of IAM-UAP on ViTs with a moderate attacking rate. In addition, we also disclose that the patch size of ViTs is a latent factor influencing the robustness against the universal attack.