Bert is Robust! A Case Against Word Substitution-Based Adversarial Attacks

In this work, we investigate the robustness of BERT using four word substitution-based attacks. We combine a human evaluation of individual word substitutions and probabilistic analysis to show that most of the adversarial examples from the four studied attacks do not preserve enough semantics from the original examples, and can thus be easily recognized by human annotators. To further confirm that, we introduce an efficient adversarial defense consisting of a data augmentation step and a post-processing step. We show that many successful attacks can be defended using our defense method by including data similar to adversarial examples during training.