Reporting guidelines for simulation-based studies in software engineering

Background: Some scientific fields, such as automobile, drugs discovery or engineer have used simulation-based studies (SBS) to faster the observation of phenomena and evolve knowledge. All of them organize their working structure to perform computerized experiments based on explicit research protocols and evidence. The benefits have been many and great advancements are continuously obtained for the society. However, could the same approach be observed in Software Engineering (SE)? Are there research protocols and evidence based models available in SE for supporting SBS? Are the studies reports good enough to support their understanding and replication? AIM: To characterize SBS in SE and organize a set of reporting guidelines aiming at improving SBS' understandability, replicability, generalization and validity. METHOD: To undertake a secondary study to characterize SBS. Besides, to assess the quality of reports to understand the usually reported information regarding SBS. RESULTS: From 108 selected papers, it has been observed several relevant initiatives regarding SBS in software engineering. However, most of the reports lack information concerned with the research protocol, simulation model building and evaluation, used data, among others. SBS results are usually specific, making their generalization and comparison hard. No reporting standard has been observed. CONCLUSIONS: Advancements can be observed in SBS in Software Engineering. However, the lack of reporting consistency can reduce understandability, replicability, generalization and compromise their validity. Therefore, an initial set of guidelines is proposed aiming at improving SBS report quality. Further evaluation must be accomplished to assess the guidelines feasibility when used to report SBS in Software Engineering.