PE-Attack: On the Universal Positional Embedding Vulnerability in Transformer-Based Models

The Transformer model has gained significant recognition for its remarkable computational capabilities and versatility, positioning itself as a fundamental component in numerous practical applications. However, the robustness of the Transformer model, specifically its stability and reliability under various types of adversarial attacks, is of utmost importance for its practical applicability. Furthermore, it offers valuable insights for the design of more efficient and secure models. In contrast with conventional investigations into adversarial robustness, our study focuses on the analysis of Positional Embeddings (PEs), a crucial component that sets the Transformer model apart from previous model architectures. Theoretical analysis of PEs has been limited due to previous predominantly empirical design, which includes features such as sinusoidal or linear patterns, learned or fixed characteristics, and absolute or relative measurements. Our investigation delves deep into potential vulnerabilities within PEs. Initially, we develop a set of input infection techniques that can be universally applied to exploit vulnerabilities present in the Transformer architecture and its variants. In addition, we propose a novel adversarial attack that manipulates the model by providing it with incorrect positional information, enabling an evasion attack. Significantly, in contrast to previous attacks that were limited to a single task, our conducted experiments involving time-series analysis, natural language processing, and computer vision indicate that the susceptibility of PEs could be universal and transferable. This finding serves as a significant warning for future Transformer-based model design, urging researchers to consider potential security risks inherent in the model’s structure.