A Corpus for Measuring Retrieval Performance of Process Matching Methods

Business process management manages the business processes of any organization thus helps in improving its performance. In any organization, the number of process models in the repository depends upon their size. To search process model from such repositories effective search techniques are required. Searching techniques require similarity computation of query and source pairs. For this purpose, several techniques have been developed. However, the developed techniques have not been evaluated rigorously or evaluated by using few query pairs that do not contain the necessary diversity to challenge the ability of these approaches. This limited evaluation is due to the absence of benchmark queries and their corresponding process models, as evaluated by domain experts. It is thus required to generate a necessary and sufficient number of queries and their corresponding source pairs. To address these two issues, in this study 669 process models as source collection has been considered and two contributions have been made, first is the use of the systematic procedure to identify a necessary and sufficient set of queries, and second being generation of human annotations related to the query models. For the generation of human annotation 2800 manual comparisons were performed by two annotators. We believe that the benchmark contains necessary diversity to evaluate the capabilities of process matching techniques.