On the evaluation of automatic program repair techniques and tools

Since 2009 several scalable and promising techniques to automatic program repair have been proposed with each technique often accompanied with a prototype tool. These techniques work in different levels of code with various types of defects and designed for different programming languages. Now the subfield of automatic program repair is mature enough to merit evaluate existing techniques and tools. This evaluation helps us identify the strengths and weaknesses of current research and provides future direction. To this end, in this paper, we present a family of criteria grouped into seven sets for evaluating automatic program repair techniques and tools. Moreover, a five-level maturity model is proposed for the mentioned subfield. To the best of our knowledge, no research yet evaluates automatic program repair techniques and tools in a general, broad, and comprehensive manner and this is the first attempt towards this goal. We employed our criteria to three existing mutation-based techniques and their corresponding tools and reported the results of preliminary evaluation. The proof-of-concept results demonstrate different aspects, capabilities and shortcomings of each technique and provide evidence to the applicability and utility of our criteria.