Using Data Flow-Based Coverage Criteria for Black-Box Integration Testing of Distributed Software Systems

Modern automotive E/E systems are implemented as distributed real-time software systems. The constantly growing complexity of safety-relevant software functions leads to an increased importance of testing during system integration of such systems. Systematic metrics are required to guide the testing process during system integration by providing coverage measures and stopping criteria but few studied approaches exist. For this purpose, we introduce a data-flow based observation scheme which captures the interplay behavior of involved ECUs during test execution and failure occurrences. In addition, we introduce a data flow-based coverage criterion designed for black box integration. By applying the observation scheme to test cases and associated faults found during execution, we first analyze similarities in data flow coverage. By further analyzing the data flow of failures, that slipped through the phase of system integration testing, we evaluate the usefulness of test gaps identified by using the suggested coverage criterion. We found major differences in the usage of data flow between undetected failures and existing test cases. In addition, we found that for the studied system under test the occurrence of failures is not necessarily a direct consequence of the test execution due to functional dependencies and side effects. Overall, these findings highlight the potential and limitations of data flow-based measures to be formalized as coverage or stopping criteria for the integration testing of distributed software systems.