MVGText: Momentum and Variance Guided Hard-Label Text Test Case Generation

Deep neural network (DNN) based text intelligence softwares have proliferated and have been widely used. However, they face challenges such as robustness. Testing is essential to DNNs in real-world applications. Researchers have proposed various testing techniques for generating test cases. With the black-box hard-label setup where only the output label of the DNN is accessible, related studies mainly concentrated on generating the new text test case by iteratively perturbing single initial test case, which limits the effectiveness. To address this issue, this paper proposes a new text test case generation method MVGText (Momentum and Variance Guided hard-label Text test case generation) for text-oriented DNN, which generates high-quality text test cases by searching based on multiple initial test cases. Firstly, multiple initial text test cases are generated randomly, in which global fixed word replacement is employed to enhance the success rate. Then, momentum and variance are applied to guide the search for the more imperceptible test case. Finally, greedy optimization is used to obtain a higher quality test case. Experimental evaluation shows that MVGText achieves an average post-test success rate of 1.4%. Furthermore, the test cases generated by MVGText exhibit the highest similarity to the original text compared to other comparative methods.