A Systematic Mapping Study on Robotic Testing of Mobile Devices

Context: Test automation is often seen as a possible solution to overcome the challenges of testing mobile devices. However, most of the automation techniques adopted for mobile testing are intrusive and, sometimes, unrealistic. One possible solution for coping with intrusive and unrealistic testing is the use of robots. Despite the growing interest in the intersection between robotics and software testing, the motivations, the usefulness, and the return of investment of adopting robots for supporting testing activities are not clear. Objective: We aim at surveying the literature on the use of robotics for supporting mobile testing with a focus on the motivations, the types of tests that are automated, and the reported effectiveness/efficiency. Method: We conduct a systematic mapping study on robotic testing of mobile devices (hereafter, referred as robotic mobile testing). We searched primary studies published since 2000 by querying five digital libraries, and by performing backward and forward snowballing cycles. Results: We started with a set of 1353 papers and after applying our study protocol, we selected a final set of 20 primary studies. We provide both a quantitative analysis, and a qualitative evaluation of the motivations, types of tests automated and the effectiveness/efficiency reported by the selected studies. Conclusions: Based on the selected studies, allowing more realistic interactions is among the main motivations for adopting robotic mobile testing. The tests automated with the support of robots are usually system-level tests targeting stress, interface, and performance testing. More empirical evidence is needed for supporting the claimed benefits. Most of the surveyed work do not compare the effectiveness and efficiency of the proposed robotics-based approach against traditional automation techniques. We discuss the implications of our findings for researchers and practitioners, and outline a research agenda.

I. Introduction

Software testing is a fundamental aspect of software engineering as it ultimately contributes to improve the quality of software systems. Mobile testing refers to the application of software testing techniques to the context of mobile devices (e.g., smartphones and tablets). Given the pervasiveness of such devices in our daily lives — 1.38 billion smartphones are expected to be shipped this year, and an even bigger number is expected for 2022 [1] — guaranteeing the quality of the delivered software is of utmost importance.