AI vs. AI: Comparing Artificial Intelligence with Actual Intelligence for a Gameplay Task

Is AI always the best choice for every task? We conducted two user studies, one large-scale and one small-scale, to attempt a testing-related gameplay task, where the goal was to maximize the code and data coverage of a set of Atari 2600 games. This particular problem was previously addressed using an AI-based system published by Ganesh et al. in the 2023 IEEE Conference on Games. Our new, human-derived results not only replicate the coverage shown in the previous AI-based study, but more importantly, by using humans for gameplay we were able to get those results with much less time, effort, and resources.