Experimentation in Software Ecosystems: a Systematic Literature Review

Context: Software ecosystems have transformed many industries, redefining collaboration and value co-creation. The success of such ecosystems depends on the dynamism of the network of users on its different sides. Consequently, decision-making in such multifaceted and interconnected environments is more complex than in conventional software products. On-line controlled experiments are considered the gold standard for aiding decision-making in software engineering processes. Experiments are extensively used to reduce bias and estimation noise for design, engineering, and business decisions. However, experimentation in software ecosystems is inherently more com-plex as it deals with atypical sources of bias and technical complications. Primary studies of experimentation approaches in software ecosystems are scattered across multiple domains and disciplines, and secondary research on the topic is scarce as highlighted in different tertiary studies. Hence, we conducted this study. Objectives: To explore primary research on experimentation in software ecosystems; Summarize current approaches, toolboxes, and solutions that practitioners and researchers, facing similar problems, can use to inform their approaches; To outline underexplored research areas and provide recommendations for practitioners. Method: We conducted a systematic literature review. The search strategy, application of exclusion and inclusion criteria, and subsequent quality assessment resulted in 63 relevant studies. Data extraction process was designed and carried out to collect data relevant to the study objectives. The extracted data under-went descriptive and thematic syntheses and analyses, in addition to cross-analysis on relevant axes. Contributions: The study resulted in four contributions. First, a distillation of the themes and patterns in the available research on the topic. Second, a practical summary of the experimental designs specific to each software ecosystem type. Third, an actionable road map for practitioners in order to achieve exper-imentation maturity in software ecosystems. Fourth, an outline of the underexplored research areas.