A Simple Statistical Approach to Fix Effectiveness Demonstration for Eliminated Mechanisms

A simple fix-effectiveness test methodology is described for issues that have been deemed completely removed. As with any statistical demonstration there is a risk that the test could pass when the issue is not resolved. The problem is particularly difficult in the case of expected failure elimination since a statistically significant improvement could cause the test to pass even though the issue has not been fully resolved, however, simple rules regarding the fix are applied in order to partially mitigate this risk. Although it is possible to use both the upper bound of the failing distribution along with the lower bound of the fixed distribution to take into account variation of both data sets, the data suggests that this added complication is not necessary, allowing for a simplified approach using the nominal fit to the failing distribution and the lower bound of the zero failure fixed distribution. Small numbers of failures can cause optimistic results which can be countered by increasing the confidence in the fix-effectiveness demonstration. The method described here is simple and makes it easy to understand and implement by the non-practitioner helping to ensure adoption by partners. This helps ensure a reasonable approach to fix-effectiveness is followed outside of the quality/reliability community therefore minimizing the risk of false fixes entering the qualification phase. Rules of thumb, such as 3X time to failure testing, make tacit assumptions that, if not fully comprehended, will give incorrect and misleading results. The method described here explicitly pulls these assumptions out into the open and allows for their comprehension.