MGBS: Multi-granularity Beam Search for NMT Model Attack

Exploring high quality adversarial example is significant in understanding neural machine translation (NMT) models’ behavior and improving their robustness. Various NMT attack methods have been proposed, which manipulate the input text on the character, word, or phrase levels following a predefined static sequence. However, manipulating text on a certain granularity easily results in incoherent samples, and the prefixed attacking order is hard to yield a high attack success rate. In this paper, we devise a Multi-granularity Beam Search (MGBS) approach for black-box NMT model attack. Specifically, our MGBS has three major merits. First, it attacks NMT models on both word and phrase levels to improve the fluency and readability of adversarial sentences. Second, we present a dynamic beam search strategy to adaptively determine the attack sequence with the least perturbation cost. Third, we designs a more reasonable and practical objective function to comprehensively evaluate the semantic similarity and BLEU scores of the output items. To evaluate the effectiveness of our proposed MGBS algorithm, we attack two popular NMT models, i.e., T5 and BART, on four translation tasks from three widely used data sets, including TED, WMT14, and WMT16. Experimental results demonstrate that our MGBS algorithm achieves the highest attack success rate with the least word modifications compared with baselines.