A Convenient Deep Learning Model Attack and Defense Evaluation Analysis Platform

Researchers have been studying in recent years how to improve the security of deep learning models to resist adversarial attacks. However, attack and defense algorithms are generally targeted, a unified, comprehensive, efficient, and convenient analysis platform is needed for deep learning model security evaluation. Deep learning model security evaluation faces the following challenges:(i) The difference of deep learning frameworks used to generate models causes inconvenience in model security evaluation. (ii) Most of the attack and defense algorithms are not universally applicable. A comprehensive measurement baseline is needed to comprehensively evaluate various attack and defense algorithms. (iii) A set of quantitative metric system needs to be proposed. In this paper, a convenient deep learning model attack and defense evaluation analysis platform is designed to automate the evaluate process. Users need only a small amount of configuration to complete the evaluation of various attack and defense algorithms. The platform integrates a variety of black-box attack algorithms and white-box attack algorithms and 6 defense algorithms. Besides, a system of metrics is constructed, covering imperceptibility, robustness, attack efficiency, etc. Based on this platform, we evaluate the performance of various algorithms on common image classification models. Compared with other platforms, this platform is efficient and convenient for evaluation, and has a wide range of evaluation. It has the ability of comparing and analyzing models generated by various frameworks and is useful for research on the security of deep learning models.