Regression Test Selection of Manual System Tests in Practice

Regression testing analyzes whether software maintenance has inadvertently broken existing functionality. Since it is costly - especially for manual testing - it is typically limited to a subset of test cases. Since impact analysis of code modifications on test cases is far from trivial for real world software, regression test selection is hard. However, if it misses affected test cases, bugs may remain unnoticed. In response, the research community has proposed numerous test selection approaches. Regression test selection is especially relevant for manual tests, since their execution costs limit the number of tests that can be executed in practice. However, evaluations of existing work focus on automated tests. Its applicability to manual tests is thus unclear. We present an industrial case study that demonstrates the challenges that regression test selection techniques face when applied to manual system tests. Furthermore, we sketch how, given these challenges, manual regression test selection can be improved.