A Practical Evaluation of UI Element Detection for Automated Mobile Game Testing

Developing attractive mobile games requires extensive testing to ensure the quality and reliability of their rich graphics and user interface (UI). Frequent UI changes during development necessitate vision-based testing; however, detecting UI elements in mobile games has not been fully explored compared to web pages and mobile apps. In this study, we benchmark recent UI detection methods using our newly constructed dataset, which comprises three different types of mobile games. We compare the methods with several evaluation metrics, including three proposed metrics designed for automated testing. The experimental results show that existing methods still face difficulty in accurately detecting UI elements, yet they could be practically used for specific purposes, such as monkey testing.