Improve Adversarial Robustness via Probabilistic Distributions Decoupled Network While Guaranteeing Clean Performance

The deep neural network (DNN) can be fooled by introducing a small perturbation to the example. Adversarial training (AT) is proven to be one of the most effective methods to improve robustness. Although the latest works to improve AT can ensure the adversarial robustness of PGD, the exploration of the performance under more disturbances is lacking, which may lead to our misunderstanding of model generalization. Moreover, they sacrifice DNN’s performance of classifying clean examples, which is called clean performance. Research shows that overfitting will be caused by the cross-mixing of distributions between natural and adversarial examples. We discover that the primary cause of cross-mixing is using the same batch-normalization (BN) layer to normalize the natural and adversarial examples because their means and variances are substantially different. We propose a new defense method which is called probabilistic distributions alignment adversarial training (PDAT). It utilizes the model with a multi-BN structure, which contains natural BN (NBN) and adversarial BN (ABN) to normalize the natural and adversarial examples separately and calculate their corresponding probabilistic distributions. It mutually trains the model through the alignment of probabilistic distributions between NBN and ABN. The natural training (NT) process of NBN is utilized to guide the training process of ABN which is much more difficult. To ensure that the model learns the correct knowledge during the alignment process, we use dynamic weights and soft-decision schemes in the loss function. This alleviates the cross-mixing problem. We conduct experiments on CIFAR10 and CIFAR100 and verify that PDAT improves robustness across a wide range of attacks while maximally maintaining clean performance. This proves that PDAT generalizes better.