RoScript: A Visual Script Driven Truly Non-Intrusive Robotic Testing System for Touch Screen Applications

Existing intrusive test automation techniques for touch screen applications (e.g., Appium and Sikuli) are difficult to work on many closed or uncommon systems, such as a GoPro. Being non-intrusive can largely extend the application scope of the test automation techniques. To this end, this paper presents RoScript, a truly non- intrusive test-script-driven robotic testing system for test automation of touch screen applications. RoScript leverages visual test scripts to express GUI actions on a touch screen application and uses a physical robot to drive automated test execution. To reduce the test script creation cost, a non-intrusive computer vision based technique is also introduced in RoScript to automatically record touch screen actions into test scripts from videos of human actions on the device under test. RoScript is applicable to touch screen applications running on almost arbitrary platforms, whatever the underlying operating systems or GUI frameworks are. We conducted experiments applying it to automate the testing of 21 touch screen applications on 6 different devices. The results show that RoScript is highly usable. In the experiments, it successfully automated 104 test scenarios containing over 650 different GUI actions on the subject applications. RoScript accurately performed GUI actions on over 90% of the test script executions and accurately recorded about 85% of human screen click actions into test code.