WallMauer: Robust Code Coverage Instrumentation for Android Apps

Code coverage is the primary metric used to assess the quality of test suites, and it is the foundation of many automated techniques ranging from fault localization to search-based optimization approaches. Code coverage is measured by inserting probes into programs which keep track of executed code when running tests. While this can be easily done in many testing domains, it remains a challenging task for Android apps, mainly due to the nature of the Dalvik bytecode used for Android apps: First, the internal handling of registers inhibits common types of probes. To circumvent this problem, existing tools often rely on conversion of Dalvik bytecode to standard Java bytecode or source code, but during the conversion back to Dalvik bytecode errors and inconsistencies may occur Furthermore, a strict limit of the number of methods and classes contained in a single archive of Dalvik bytecode ($D E X$ file) requires spliting apps into multiple such DEX files (multidex approach), which is rarely supported by existing coverage instrumentation frameworks. This is not only a problem when trying to instrument regular multidex apps, but the coverage instrumentation itself increases the number of methods, potentially requiring a multidex solution even for apps that would otherwise fit in a single DEX file. In this paper we present WALLMAUER, a new code coverage tool that overcomes these limitations: It supports multidex, and avoids inconsistencies by rigorously instrumenting Dalvik bytecode directly. WALLMAUER solely requires an APK file as input and as such it can be easily integrated into any existing testing environment. Using a set of 1000 open source apps from the F-Droid repository we demonstrate that WALLMAUER is extremely robust, successfully instrumenting more than 99% of apps, more than any other state-of-the-art instrumentation framework.