Leveraging Retrieval Augmented Generation to Enhance LLM-Based Fault Localization for Novice Programs

Fault localization (FL) in novice programs is critical for computer science education, yet it remains insufficiently explored compared to industrial programs. Traditional FL methods such as Spectrum-Based Fault Localization (SBFL) and Mutation-Based Fault Localization (MBFL) primarily rely on test coverage or mutation analysis, exhibiting significant limitations when handling novice programs. In recent years, Large Language Models (LLMs) have enabled more effective fault localization through semantic code analysis. However, existing LLMs primarily rely on internal knowledge acquired during training and lack access to specialized external knowledge bases, particularly limiting their effectiveness when addressing novice programs with distinctive fault patterns. To address these limitations, we propose NPFL-RAG, a LLM-based fault localization framework that combines Retrieval Augmented Generation (RAG). This framework regards the FL task as a three-step process: Knowledge Base Construction, which builds a comprehensive repository containing multidimensional fault-fixing information; Relevant Fix Cases Identification, which employs multidimensional retrieval and Retrieval Results Optimization (RRO) mechanism to identify the most relevant fix cases; and Fault Localization Result Generation, which integrates the retrieved knowledge with the LLM's inherent understanding to localize potential faults. We conduct a comprehensive experimental evaluation on the TutorCode dataset. The results demonstrate that NPFL-RAG significantly outperforms baselines across multiple evaluation metrics. In particular, when incorporating the RAG mechanism, DeepSeek-v3 achieves a 7.1 % improvement in the TOP-1 metric compared to the without RAG technique and performs approximately 7 times better than traditional MBFL techniques. Furthermore, we confirm the indispensability of the components in NPFL-RAG with the ablation study and demonstrate the usability of RRO mechanism in fault localization and explanation tasks.