Empirical Evaluation of Large Language Models for Novice Program Fault Localization

Integrating Large Language Models (LLMs) into software fault localization represents a significant advancement in improving debugging efficiency for programmers. However, novice program fault localization, which is essential for computer science education, has not been thoroughly investigated in previous studies. In contrast to industrial programs target practical functionality, novice programs primarily deal with individual algorithmic issues. The distinct logic structures between novice and industrial programs can impact how effectively LLM understand and process them. Moreover, this difference reveals the inapplicability of the Competent Programmer Hypothesis, a fundamental assumption in industrial fault localization, to novice program fault localization. Therefore, industrial methodologies are unsuitable for novice programming, emphasizing the need for our empirical studies. To fill this gap, we evaluate LLMs’ effectiveness in localizing faults for novice programs in statement level. Using the widely used novice programs dataset Codeflaws and Condefects, we compare the performance of two commercial LLMs (i.e., ChatGPT-3.5 and ChatGPT-4) and three open-source LLMs (i.e., ChatGLM3, Llama2, and Code Llama) against traditional fault localization methods, examining their accuracy and overlap. Additionally, we investigate how prompt engineering improves localization precision. Our findings show ChatGPT-4’s overall superior performance, with ChatGPT-3.5 exhibiting minor advantages in certain cases. ChatGPT-4 outperforms the traditional methods with best performance by 592% and 137% on Codeflaws and Condefects. Specifically, each method exhibits unique strengths in localizing novice programming faults. Moreover, carefully crafted prompts can improve LLMs’ precision. These insights underscore the promising potential of utilizing LLMs for fault localization in novice programming.