A Comparative Analysis of Test Automation Frameworks Performance for Functional Testing in Android-Based Applications using the Distance to the Ideal Alternative Method

The importance of functional testing is to guarantee the quality of an Android application to enhance the user experience so as to be more secure and reliable under market pressure, forcing many companies to build high quality and high performance applications that can be released in a short time. Thus, the testing process should be automated to make sure that applications run quickly and effectively. There are several test automation frameworks available, specifically designed for Android applications which offer features and efficient performance to test applications more effectively, such as Espresso, Calabash and Appium. The purpose of this research is to find the best framework based on performance with certain parameters and sub-parameters. The authors ran function tests on each framework with several test cases and provisions from each framework in an Android-based application. Reports were generated from each of the frameworks to collect data in the form of test cases covered, time complexity, execution speed, and element inspection for automated testing progress parameters. These were compared with a matrix that defines tool usability parameters using The Distance to the Ideal Alternative method (DIA) with a weight for each parameter and sub-parameter. The results show that Calabash has the best performance among the other frameworks with a 0 value for the DIA Method that indicates that Calabash has zero distance to the ideal alternative.