Worldviews, Research Methods, and their Relationship to Validity in Empirical Software Engineering Research

Background - Validity threats should be considered and consistently reported to judge the value of an empirical software engineering research study. The relevance of specific threats for a particular research study depends on the worldview or philosophical worldview of the researchers of the study. Problem/Gap - In software engineering, different categorizations exist, which leads to inconsistent reporting and consideration of threats. Contribution - In this paper, we relate different worldviews to software engineering research methods, identify generic categories for validity threats, and provide a categorization of validity threats with respect to their relevance for different world views. Thereafter, we provide a checklist aiding researchers in identifying relevant threats. Method - Different threat categorizations and threats have been identified in literature, and are reflected on in relation to software engineering research. Results - Software engineering is dominated by the pragmatist worldviews, and therefore use multiple methods in research. Maxwell's categorization of validity threats has been chosen as very suitable for reporting validity threats in software engineering research. Conclusion - We recommend to follow a checklist approach, and reporting first the philosophical worldview of the researcher when doing the research, the research methods and all threats relevant, including open, reduced, and mitigated threats.