BDTest: A Diversity-Oriented Test Case Generation Framework for Deep Neural Networks in 6G-IOT

The widespread integration of Artificial Intelligence (AI) in sixth-generation Internet of Things (6G-IoT) applications, introduces significant challenges for ensuring the trustworthy and dependability of AI models. The "black-box" characteristic of numerous Deep Neural Networks (DNNs) creates a notable obstacle for confirming their safety in intricate, ever-changing environments. Consequently, there is a need for extensive testing, requiring the gathering and labeling of a large number of test cases, a process that is both time-intensive and resource-consuming. While previous studies have adapted neuron coverage criteria for steering test case generation in DNNs. Yet, these criteria are white-box measures requiring access to model states and presenting their practical limitations. Conversely, black-box metrics, which focus on outputs, present a more feasible approach. Among these, black-box diversity metrics evaluate model robustness by generating diverse test cases, eliminating the need for internal model details. This paper presents a test case generation framework centered on diversity, known as BDTest. BDTest enhances test adequacy through five stages: (1) Mapping feature vectors extracted from an initial set of seed images onto a low-dimensional manifold utilizing UMAP; (2) Detecting sparse regions using DBSCAN; (3) Sampling key points from these regions via Latin Hypercube Sampling; (4) Reconstructing latent features and generating new images through ICA and GAN inversion; and (5) Measuring the diversity of the generated set using metrics such as the Log-Determinant. Experiments demonstrate that BDTest significantly improves test set diversity and error detection performance, achieving error rates of 59.36%, 59.76%, and 67.03% on VGG19, DenseNet121, and MobileNetV2, respectively, outperforming DeepXplore by an average of 12.43% and DLFuzz by 9.95% across all tested models. When retrained with the generated test cases, the model demonstrated improved accuracy on the original test set, alongside a significant enhancement in accuracy on the natural adversarial test set.