SPLAS: An Autonomous Lightweight and Versatile Image Classification Robustness Benchmark

Image classification is one of the most fundamental vision tasks and remarkable progress in prediction accuracy has been made with deep neural networks and vision transformers in the last decade. However, in real-world applications, these models face many challenges, especially data quality issues. Unexpected behaviors are often observed in such degraded situations. The researchers began to comprehend the robust design of the classification models. However, there is a lack of compact and comprehensive benchmarks that quickly and precisely evaluate the robustness of a model in multiple dimensions. This work proposes a lightweight and versatile robustness benchmark called SPLAS that has five components: SuperHard, PartialInfo, LowResolution, AdversarialAttack and SpeckleNoise. It contains $50\,000$ images evenly distributed over $1\,000$ classes and evaluates a classification model in five aspects: accuracy on intrinsically difficult images, images with partial information, robustness against low resolution, adversarial attacks, and speckle noise. In addition, there is no human decision intervention in the inclusion of SPLAS image. Instead, benchmark images are selected on the basis of the vote of a group of surrogate models. We tested 80 renowned and widely utilized models on SPLAS. Among them, a vision transformer, ViT-L/16, secures the best score of 51.16 out of 100 while 74 models achieve a score under 30. The whole benchmark is publicly available on Google Drive and IEEE DataPort.