Program Interoperable Large Language Model Software Testing Scheme: a Case Study on JavaScript Engine Fuzzing

Large language models (LLMs) have cultivated impressive semantics capabilities and expert knowledge from their vast pre-training corpora, especially showing prospects in automated software testing. However, LLMs are designed for human interaction, which poses the following challenges when interacting with programs for testing: 1) LLMs cannot communicate directly with programs, and there is no existing paradigm to establish interaction between them. 2) Existing evaluation methods are unable to assess the quality of LLMgenerated tests during software testing. 3) Current LLM-guided testing generation cannot be optimized in real time, resulting in low testing efficiency. To address these challenges, we present PILLM, a program interoperable LLM scheme. First, we designed a prompt mechanism for interactive program testing based on the source code semantics and expert knowledge from the LLM. Second, we proposed an evaluation mechanism for the PILLM's test code generation, thus obtaining test seeds that are semantically related to the corresponding source code. Third, PILLM optimizes the next generated tests based on the coverage and source code execution information obtained in the program execution. In the 24-hour running experiment, PILLM improved the coverage by 45.7% and 14.9% compared to Fuzz4all and Fuzzilli respectively. PILLM proves its effectiveness by finding four new real-world bugs in the JavaScript engine. We have released the source code of PILLM as open source on Github.