QAQA-SS: An Improved Fuzzing Approach with Seed Scheduling Based on the UCB Algorithm for QA Systems

Question Answering (QA) system is a type of artificial intelligence application designed to comprehend natural language questions and provide accurate, meaningful natural language answers. However, there are errors in the QA system, which can cause inconvenience for users and even lead to economic losses and security issues. In response to these challenges, QAQA showed up. QAQA designs several metamorphic relations (MRs) and achieves precise automatic testing through semantic-guided search and enhanced test oracles. However, QAQA fairly selects each seed (initial test input) in sequential order and assigns them a fixed number of mutations, which leaves many seeds that could produce quality test inputs neglected. This paper proposes an improved version named QAQA-SS (QAQA-Seed Schedule), where we enhance QAQA with a seed scheduling module and a fuzzing framework featuring iterative feedback. We combine the Upper Confidence Bound (UCB) algorithm with perplexity to devise a novel seed scheduling algorithm. QAQA-SS selects superior seeds and allocates appropriate mutation counts (energy) to generate more error-inducing test inputs within a limited energy budget. Additionally, we introduce a text paraphrasing operation to improve the performance of finding errors in the QA system. We conducted experiments following the experimental setup of QAQA on three datasets: BoolQ, SQuAD2, and NarrativeQA. Our experimental results reveal a noteworthy enhancement in the error discovery rate for QAQA-SS, achieving 81,777 error discoveries compared to 64,354 error discoveries with QAQA.