A Survey of Methodology in Self-Adaptive Systems Research

Major research venues on autonomic and self-adaptive systems have been active for 16 years, exploring and building on the seminal vision of autonomic computing in 2003. We study the current trajectory and progress of the research field towards this vision, surveying the research questions that are asked by researchers and the methodological practice that they employ in order to answer these questions. We survey contributions under this lens across the three main venues for primary research in autonomic and self-adaptive systems work: ICAC, SASO, and SEAMS. We examine the last three years of contributions from each venue, totaling 210 publications, to gain an understanding of the dominant current research questions and methodological practice - and what this shows us about the progress of the field. Our major findings include: (i) most research questions still focus one level below the highest autonomy level vision; (ii) methodological practice is split almost evenly between real-world experiments and simulation; (iii) a high level of positive results bias exists in publications; and (iv) there are low levels of repeatability across most contributions.