Selecting “good” regression tests based on a classification of side-effects

When software systems undergo modifications, regression testing is a prominent risk mitigation technique to safeguard the quality of actually unmodified parts of that system. Being a change-based test type, regression tests should ideally be automatically executed as often as modifications happen to the system. When regression testing takes too long to be executed in its entirety, a selection has to be done to execute only those regression tests that safeguard the unmodified parts of the system the best. The potential to detect side-effects is associated to the type of modification made to the system and varies among regression tests. A selection algorithm must be able to identify “good” regression tests according to the modifications made. A “good” regression test is a regression test that has a higher capability to detect probable unwanted side-effects of a modification. This paper introduces a novel approach to regression test selection based on classification of side-effects and quantification of the side-effect detection potential of each regression test. Two approaches to regression test selection are described. Both approaches were implemented by a prototype and integrated into the CI/CD pipeline of the industrial software system Vaadin. Eventually, the effectiveness of the selection approach is evaluated.