“No Free Lunch” when using Large Language Models to Verify Self-Generated Programs

Large Language Models (LLMs) have shown great success in a wide range of text-generation tasks including the synthesis of code from natural language descriptions. As LLMbased techniques continue to grow in popularity, especially amongst entry-level developers, LLM-generated code has the potential to be deployed in a diverse set of application domains. While LLMs can generate syntactically correct code output, recent work has shown the presence of nonsensical and faulty reasoning in LLM-generated text. As such, overreliance on LLMs for software generation may potentially result in the deployment of faulty software leading to critical system failures. This study explores the capabilities of a single LLM to generate both software and corresponding test suites from the same initial program descriptions, which can be considered analogous to an individual developer coding and unit testing for a given piece of software. We present an empirical framework and evaluation methodology to assess the usefulness of LLM-generated test cases for verifying programs generated by the same LLM. Our findings indicate that LLMs frequently generate irrelevant tests that suffer from numerous quality concerns.