A Comparative Evaluation of Requirement Template Systems

Context: Multiple semi-formal syntax templates for natural language requirements foster to reduce ambiguity while preserving readability. Yet, existing studies on their effectiveness do not allow to systematically investigate quality benefits and compare different notations. Objectives: We strive for a comparative benchmark and evaluation of template systems to support practitioners in selecting template systems and enable researchers to work on pinpoint improvements and domain-specific adaptions. Methods: We conduct a comparative experiment with a control group of free-text requirements and treatment groups of their variants following different templates. We compare effects on metrics systematically derived from quality guidelines. Results: We present a benchmark consisting of a systematically derived metric suite over seven relevant quality categories and a dataset of 1764 requirements, comprising 249 free-text forms from five projects and variants in five template systems. We evaluate effects in comparison to free text. Except for one template system, all have solely positive effects in all categories. Conclusions: The proposed benchmark enables the identification of the relative strengths and weaknesses of different template systems. Results show that templates can generally improve quality compared to free text. Although MASTER leads the field, there is no conclusive favourite choice, as overall effect sizes are relatively similar.