Comprehensive Evaluation of an Educational Software Engineering Simulation Environment

Software engineering educational approaches are often evaluated only anecdotally, or in informal pilot studies. We describe a more comprehensive approach to evaluating a software engineering educational technique (SimSE, a graphical, interactive, customizable, game-based software engineering simulation environment). Our method for evaluating SimSE went above and beyond anecdotal experience and approached evaluation from a number of different angles through a family of studies designed to assess SimSE's effectiveness and guide its development. In this paper, we demonstrate the insights and lessons that can be gained when using such a multi-angled evaluation approach. Our hope is that, from this paper, educators will: (1) learn ideas about how to more comprehensively evaluate their own approaches, and (2) be provided with evidence about the educational effectiveness of SimSE.