Boosting the Cost-Effectiveness of Metamorphic Test Case Pair Selection for DNN Testing with Surrogate Model

With its ability to alleviate the test oracle problem, Metamorphic Testing (MT) has been widely used to test Deep Neuron Networks (DNN). To improve failure detection ability of MT, recently, researchers have proposed uncertainty based methods to select Metamorphic test case Pairs (MPs) that are more likely to violate metamorphic relations. However, in these methods, the DNN under test needs to be frequently invoked to obtain the output probabilities of test cases for uncertainty calculation, potentially limiting their adoptions in resource-constrained test scenarios where the number of DNN calls should be minimized. To further boost the costeffectiveness of MT, in this paper, we propose MPSS, a black-box method that relies on a surrogate model to select failure-revealing MPs. In particular, MPSS aims to train and iteratively optimize a support vector machine to approximate the DNN classification boundaries in the latent space. Then, by analyzing the relative positions of both source and followup test cases of each MP to such boundaries, MPSS can effectively estimate whether the execution of this MP will lead to a metamorphic relation violation without actually calling the DNN model. Experimental results show that MPSS can increase the cost-effectiveness of MP selection by maximizing detected failures while minimizing DNN calling times under given test budgets in various situations.