Investigating Execution Trace Embedding for Test Case Prioritization

Most automated software testing tasks, such as test case generation, selection, and prioritization, can benefit from an abstract representation of test cases. Although test case representation usually is not explicitly discussed in software literature, but traditionally test cases are mostly represented based on what they cover in source code (e.g., which statements or branches), which is in fact an abstract representation. In this paper, we hypothesize that execution traces of test cases, as representations of their behaviour, can be leveraged to better encode test cases compared to code-based coverage information, for automated testing tasks. To validate this hypothesis, we propose an embedding approach, Test2Vec, based on an state-of-the-art neural program embedding (CodeBert), where the encoder maps test execution traces, i.e. sequences of method calls with their inputs and return values, to fixed-length, numerical vectors. We evaluate this representation in automated test case prioritization (TP) task. Our TP method is a classifier trained on the passing and failing vectors of historical test cases, in regression testing. We compare our embedding with multiple baselines and related work including CodeBert itself. The empirical study is based on 250 real faults and 703,353 seeded faults (mutants) over 250 revisions of 10 open-source Java projects from Defects4J, with a total of over 1,407,206 execution traces. Results show that our approach improves all alternatives, significantly, with respect to studied metrics. We also show that both inputs and outputs of a method are important elements of the execution-based embedding.

1. Introduction

Code coverage is one of the most common test adequacy metrics [1], which is used to optimize tasks such as automated test generation [2] and prioritization [3]. However, the common code-based coverage metrics (e.g., statement and branch coverage) are weak criteria in evaluating a test case/suite in terms of detecting bugs. A covered statement in the source code is not necessarily bug free. Depending on which path the execution has taken to land on that statement, the behaviour of the system might be right (passing test case) or wrong (failing test case). This is referred to as path coverage. However, all-path coverage is usually not feasible, given that number of concrete execution paths is often too large (or even infinite). Recent studies propose to represent a test case with its entire execution path, abstracted at some level (to make it feasible), but still including the order of execution (e.g., as a sequence of method calls, the test covers at run-time) [4], [5]. To make the sequence length manageable, these approaches either cut-off the trace at a fix length or map the traces into a fixed-size vector space, for instance using a one-hot encoding approach [6] over all existing methods in that code base. In this paper, we propose a novel methodology to represent a test case as an abstract execution path. We then show how we can leverage such representation to prioritize test cases which outperforms existing test prioritization methods based on common code coverage metrics as well as state of the art alternatives to represent execution traces.