On effectiveness of pairwise methodology for testing network-centric software

Pairwise testing, which can be complemented with partial or full N-wise testing, is a technique which guarantees that all important parametric value pairs are included in a test suite. A percentage of N-wise testing is also included. We conjecture that N-wise enhanced pairwise testing can be used as a black-boxed testing method to increase effectiveness of random testing in exposing unusual or unexpected behaviors, such as security failures in network-centric software. This testing can also be quite cost-efficient since small N test suites grow linearly with the number of parameters. This paper explains the results of random testing of a simulation in which about 20% of the defects with probabilities of occurrence less than 50% are never exposed. This supports the premise that if the unusual or unexpected behaviors are based on defects which are less likely to occur, then random testing needs to be enhanced, especially if those unexposed defects could cause erratic or even critical behaviors to the system. Higher system complexities may indicate higher numbers of unusual or unexpected behaviors. It may be difficult to use the traditional operational profile information to determine the amount of testing for unusual behaviors since the operational usage may be 0 or close to it. Another interesting problem is that some testers lack the experience necessary to effectively analyze the results of a test run. It is important to compensate for the lack of experience so that novice testers are able to test comparatively as effectively as more experienced testers. It is believed that if the size of the test suite is relatively small, then it may be easier to pinpoint the source of a failure. The research presented in this paper is aimed at addressing some of these issues of random testing via enhanced pairwise testing and N-wise testing in general. It is possible that more complex systems, such as those that rely a great deal on a network, would require higher numbers of interactions to combat unexpected combinations for use in some testing instances such as security testing or high assurance testing. A tool is being developed concurrently to help automate a part of the test generation process.