Bug Hide-and-Seek: An Educational Game for Investigating Verification Accuracy in Software Tests

This Innovative Practice Full Paper describes a pedagogical technique for introducing unit testing within software engineering courses. The Bug Hide-and-Seek educational game reinforces testing principles by requiring students to develop some correct solutions as well as some other solutions that intentionally contain bugs. While developing the correct and buggy solutions, students also write corresponding tests that should identify whether each solution contains bugs or exhibits acceptable behavior. Consequently, the first goal of the game is to hide a clever bug that will trick other students' tests into passing the implementation, despite the hidden bug. The second goal is to write thorough tests that can accurately differentiate correct from incorrect software behavior. We introduce the motivation, pedagogy, and preliminary analysis of two variations of the Bug Hide-and-Seek game, while comparing their tradeoffs. The between-subject variation considers each student's test suite in its entirety. The within-subject variation requires more sophisticated analysis, but considers each individual function along with its corresponding tests, which provides more granular insight and specific feedback to students. We conducted a pilot study of both variations over two semesters of students (n=87) playing the Bug Hide-and-Seek game. We found that students' test True Positive Rate and True Negative Rate at verifying implementations are both significant predictors of a lack of bugs in their own solution.