IR in Software Traceability: From a Bird's Eye View

Background. Several researchers have proposed creating after-the-fact structure among software artifacts using trace recovery based on Information Retrieval (IR). Due to significant variation points in previous studies, results are not easily aggregated. Aim. We aim at an overview picture of the outcome of previous evaluations. Method. Based on a systematic mapping study, we perform a synthesis of published research. Results. Our synthesis shows that there are no empirical evidence that any IR model outperforms another model consistently. We also display a strong dependency between the Precision and Recall (P-R) values and the input datasets. Finally, our mapping of P-R values on the possible output space highlights the difficulty of recovering accurate trace links using naïve cut-off strategies. Conclusion. Based on our findings, we stress the need for empirical evaluations beyond the basic P-R 'race'.