Benchmark Fuzzing for Android Taint Analyses

Benchmarking is the most often used technique to empirically evaluate software. To do so, benchmarks are often manually created when they are needed. Mainly two kinds of benchmarks are frequently employed: micro and real-world benchmarks. While micro benchmarks are most of the time handcrafted from scratch, real-world benchmarks are typically created by collecting available software from repositories or markets. Both types have their deficits. On the one hand, a handcrafted micro benchmark can only be of limited complexity, but the creator knows its ground-truth which is needed for precise evaluations. On the other hand, in case of a complex real-world benchmark, a ground-truth is unavailable in most cases. To bring together the best of both worlds we propose the concept of benchmark fuzzing, a three step procedure that allows for an automatic generation, execution and evaluation of benchmarks of configurable size and versatility. We implemented benchmark fuzzing in our novel Android taint analysis benchmark generation tool GenBenchDroid. Our evaluation performed on GenBenchDroidshows the benefits of benchmark fuzzing. We show that over-adaptation of benchmarks can broadly be decreased, scalability issues of analysis tools can be detected and combinations of analysis challenges that negatively impact analysis' accuracy can be identified. In addition, benchmark fuzzing allows to regenerate up-to-date versions of state-of-the-art micro and real-world benchmarks. Furthermore, our evaluation shows that the cost of benchmark fuzzing can be estimated and appears to be reasonable in regards of the advantages.