Do Injected Faults Cause Real Failures? A Case Study of Linux

Software fault injection (SFI) has been used to intentionally cause "failures" in software components and assess their impacts on the entire software system. A key property that SFI should satisfy is the representative ness of injected failures, the failures caused by SFI should be as close as possible to failures in the wild. If injected failures do not represent realistic failures, the measured resilience or tolerance against failures of the investigated system is not trustworthy. To the best of the authors' knowledge, the representative ness of "faults" has been investigated. However, it is an open problem whether the failures caused by injected faults represent realistic failures. In this paper, we report the preliminary results of the investigation on the representative ness of injected failures. To compare injected failures with real failures, we have collected 43,742 real crash logs of Linux from the Red Hat repository, and conducted a fault injection campaign on Linux, using SAFE, a state-of-theatre injector of software faults. In the fault injection campaign, 50,000 faults are injected to the Linux file system and 71,470 runs of a workload are executed. The crash logs generated by SFI are compared with the real Red Hat logs with respect to crash causes, crashed system calls, and crashed modules. Our preliminary results suggest that failures caused by injected faults do not represent real failures, probably because injected faults are not representative enough or because the selected workload is not realistic.