Peft: Multiline Complex Patch Correctness Assessment Based on Fine-Tuning Large Language Model with “Golden Data”

In recent years, automated program repair has garnered considerable attention owing to its potential to mitigate software maintenance costs. There remain challenges for automated program repair techniques to be widely applied in practice, where too many overfitting patches are one of the issues. Automated patch correctness assessment emerges as a crucial means to enhance the viability of recommended patches in automated program repair technology. The current automated patch correctness assessment methods exhibit suboptimal performance when evaluating multiline patches compared to their effectiveness in assessing single-line patches. This disparity arises because multiline patches involve intricate contextual semantic relationships and significant program structure modifications, such as branching and looping. Those complexities render it more challenging for neural networks to capture the latent features essential for assessing correctness. To address the above problem, this paper proposes an automated patch correctness assessment method, named Peft, which fine-tunes a large language model using “golden data” that is generated based on semantic and program structural features, resulting in a patch evaluation model. Experiments were conducted on three datasets with varying proportions of multiline complex patches. The patches were sourced from real-world applications or generated by 23 APR techniques for fixing Defects4J v1.2. The results demonstrated that Peft consistently outperformed existing methods across all datasets. Notably, when evaluating datasets composed entirely of multiline complex patches, Peft significantly outperformed the open-source baseline technique Cache (accuracy: 79.2% vs. 66.1%, F1-score: 81.9% vs. 69.5%).