LLM-BL: Large Language Models are Zero-Shot Rankers for Bug Localization

Bug localization, which aims to localize software faults specific to a bug report, is valuable for improving software developers' efficiency. This task is often formulated as an information retrieval problem, where potentially buggy files are retrieved and ranked according to their textual similarity to a target bug report. To address this task, many methods have been proposed, primarily focusing on resolving the semantic gap between natural language in bug reports and programming language in source files. Recently, Large Language Models (LLMs) have demonstrated strong capabilities in seamlessly understanding both natural language and programming language, potentially addressing the semantic gap issue through natural conversation. However, the limited context length of prompt and capability of long-context comprehension in existing LLMs make it impossible to directly load the entire codebase (e.g., thousands of code files) into LLMs' prompts to retrieve buggy code files. In this paper, we explore how to leverage existing LLMs for projectlevel bug localization without requiring additional fine-tuning and propose an LLM-based Bug Localization framework, LLM-BL. Our core idea is to enable LLMs to perform bug localization tasks using listwise ranking instructions and to avoid exceeding the context length limit by compressing the codebase through file filtering and content reduction. Specifically, LLM-BL consists of four modules: report expansion, candidate retrieval, content reduction, and LLM-based bug localization. Among them, the first three modules are used to retrieve potential buggy code files and extract bug-related information from the code files, respectively, while the last module enables the LLM to perform bug localization through listwise file ranking. We select three widely used LLMs (i.e., ChatGPT, Llama 3, CodeLlama) as the base LLMs in our framework and conduct extensive experiments on six public projects with two report types (i.e., Java and Python). Experimental results demonstrate that the two LLMs, ChatGPT and Llama 3, can more effectively understand task intentions and more accurately localize buggy files than CodeLlama. Compared to existing methods, LLM-BL achieves better localization performance without requiring any fine-tuning, in a plug-and-play manner. These results demonstrate that both ChatGPT and Llama 3 are effective zero-shot rankers for bug localization.