Effect of Time Window on the Performance of Continuous Regression Testing

Test prioritization is an effective technique used to reduce the amount of work required to support regression testing in continuous integration development. It aims at finding an optimal order of tests that can detect regressions faster, potentially increasing the frequency of software releases. Prioritization techniques based on test execution history use the results of preceding executions to determine an optimal order of regression tests in the succeeding test executions. In this paper, we investigate how can execution history be optimally used to increase the effectiveness of regression test prioritization. We analyze the effect of history time window on the fault detection effectiveness of prioritized regression tests. We report an experimental study using a data set from Cisco. The results suggest that varying the size of the window can considerably change the performance of regression testing. Our findings will potentially help developers and test teams in adjusting test prioritization techniques for achieving higher cost-effectiveness in continuous regression testing.