A Large-Scale Evaluation of a Model for the Evaluation of Games for Teaching Software Engineering

In order to adopt games for Software Engineering (SE) education effectively it is essential to obtain sound evidence on their quality. A prominent evaluation model is MEEGA (Model for the Evaluation of Educational Games), which provides a systematic support to evaluate the game's quality in terms of motivation, user experience and learning. To facilitate its application, the model provides a questionnaire for collecting data on the perception of the students after they played an educational game in a case study with a one-shot post-test design. However, in order to assure a valid feedback on the game's quality an important issue is the reliability and validity of the questionnaire. In this respect, this article presents a large-scale evaluation of the MEEGA questionnaire in terms of reliability and construct validity. The analysis is based on data collected in 43 case studies, evaluating 20 different SE games, involving a population of 723 students. Our analysis indicates that the MEEGA questionnaire can be considered reliable (Cronbach's alpha α=.915). In terms of construct validity, there exists evidence of convergent validity through an acceptable degree of correlation of almost all item pairs within each dimension. Yet, we identified a need for the re-grouping of items based on the results of a factor analysis, mainly with respect to items related to motivation and user experience. These results allow SE researchers and instructors to rely on the MEEGA questionnaire in order to evaluate SE games and, thus, contribute to their improvement and to direct an effective and efficient adoption for SE education.