Invalidator: Automated Patch Correctness Assessment Via Semantic and Syntactic Reasoning

Automated program repair (APR) faces the challenge of test overfitting, where generated patches pass validation tests but fail to generalize. Existing methods for patch assessment involve generating new tests or manual inspection, which can be time-consuming or biased. In this paper, we propose a novel technique,
Invalidator
, to automatically assess the correctness of APR-generated patches via semantic and syntactic reasoning.
Invalidator
leverages program invariants to reason about program semantics while also capturing program syntax through language semantics learned from a large code corpus using a pre-trained language model. Given a buggy program and the developer-patched program,
Invalidator
infers likely invariants on both programs. Then,
Invalidator
determines that an APR-generated patch overfits if: (1) it violates correct specifications or (2) maintains erroneous behaviors from the original buggy program. In case our approach fails to determine an overfitting patch based on invariants,
Invalidator
utilizes a trained model from labeled patches to assess patch correctness based on program syntax. The benefit of
Invalidator
is threefold. First,
Invalidator
leverages both semantic and syntactic reasoning to enhance its discriminative capability. Second,
Invalidator
does not require new test cases to be generated, but instead only relies on the current test suite and uses invariant inference to generalize program behaviors. Third,
Invalidator
is fully automated. Experimental results demonstrate that
Invalidator
outperforms existing methods in terms of Accuracy and F-measure, correctly identifying 79% of overfitting patches and detecting 23% more overfitting patches than the best baseline.