Evaluating the Repair Ability of LLM Under Different Prompt Settings

As the increase of complexity and scale of software, the number of software bugs increases as well, which causes huge financial losses and poses a great threat to our lives. To address this issue, researchers have devoted themselves to automated program repair(APR), which aims to release programmers from the time-consuming manual debugging. In the last few decades, lots of APR tools have been proposed, such as constraint-based, heuristic-based, template-based and learning-based APRs. Unfortunately, traditional APRs are confronted with numerous challenges. Neither do they deeply understand the program's semantics, nor they can fix many types of bugs in different scenarios. Recently, researchers directly applied Large Language Models (LLMs) for APR and demonstrated that directly applying state-of-the-art LLMs can already substantially outperform all existing APR techniques on many datasets. However, there is a lack of study that fully reveals which information is useful for LLM-based APR. There is neither a study that reveals how we can further increase the repair ability of large language models by better utilizing prompts. In this study, we conduct an extensive study to figure out how bug descriptions, fault locations and few-shot prompting affect the repair ability of LLM on Defects4J dataset. We found that 1) Concise bug description is critical for the model to fix bugs. 2) Fault locations can improve LLM's repair ability in most cases, but it does not always helpful. 3) Few-shot prompting is not beneficial for enhancing the understanding of LLM regarding the faulty program and we'd better not use few-shot prompting. We are fully convinced that our findings will provide not only some theory evidences but also some insights for future research.