Empirical Research in Software Architecture: How Far have We Come?

Context: Empirical research helps gain well-founded insights about phenomena. Furthermore, empirical research creates evidence for the validity of research results. Objective: We aim at assessing the state-of-practice of empirical research in software architecture. Method: We conducted a comprehensive survey based on the systematic mapping method. We included all full technical research papers published at major software architecture conferences between 1999 and 2015. Results: 17% of papers report empirical work. The number of empirical studies in software architecture has started to increase in 2005. Looking at the number of papers, empirical studies are about equally frequently used to a) evaluate newly proposed approaches and b) to explore and describe phenomena to better understand software architecture practice. Case studies and experiments are the most frequently used empirical methods. Almost half of empirical studies involve human participants. The majority of these studies involve professionals rather than students. Conclusions: Our findings are meant to stimulate researchers in the community to think about their expectations and standards of empirical research. Our results indicate that software architecture has become a more mature domain with regards to applying empirical research. However, we also found issues in research practices that could be improved (e.g., when describing study objectives and acknowledging limitations).