A Selection Method for Black Box Regression Testing with a Statistically Defined Quality Level

We consider regression testing of safety-critical systems consisting of black-box components. This scenario is common for automotive electronic systems where testing time is expensive and should be reduced without uncontrolled reduction of reliability. This requires a methodology to select test cases from a larger test suite for which a defined quality level of the resulting regression test cycle can be provided. With this in mind, we propose a method to select test cases based on a stochastic model. We are modeling test case failure probabilities as dependent random variables, therefore already observed test results have an influence on the estimation of further test case failure probabilities. Based on an information theoretical approach, we validate the mutual differential information degree between test case failure probabilities and compute a function which returns the risk for not selecting a test case, i.e., the probability of not selecting a failing test case. Depending on the mutual differential information significant reductions of testing time can be achieved while testing reliability is preserved at a quantifiable high level. We will validate theoretically our results and will show in an industrial case study the benefits of our method.