A Framework for Software Engineering Experimental Replications

Experimental replications are very important to the advancement of empirical software engineering. Replications are one of the key mechanisms to confirm previous experimental findings. They are also used to transfer experimental knowledge, to train people, and to expand a base of experimental evidence. Unfortunately, experimental replications are difficult endeavors. It is not easy to transfer experimental know-how and experimental findings. Based on our experience, this paper discusses this problem and proposes a Framework for Improving the Replication of Experiments (FIRE). The FIRE addresses knowledge sharing issues both at the intra-group (internal replications) and inter-group (external replications) levels. It encourages coordination of replications in order to facilitate knowledge transfer for lower cost, higher quality replications and more generalizable results.

1. Introduction

Empirical research is fundamental to the evolution of software engineering as a discipline, but according to Kitchenham et al. the current state of empirical research in software engineering (i.e. case studies, surveys, and formal experiments) is less than ideal [22]. Surveys of the software engineering literature show that the majority of the papers contain little or no empirical validation [27], [28], [30], [31]. This lack of empirical work is more disturbing considering that it is difficult to build a usable body of knowledge from individual studies. The results of an isolated experiment are not likely to be directly applicable to every practitioner due to differences in system domains, subject profiles and cultural environments [18]. Replications must be conducted to ensure more robust evidence sets that can support generally applicable conclusions [5]. These replications can be conducted by the same research group (i.e. internal replications) or by different researchers, in different contexts (i.e. external replications).