Utilizing design of experiments to reduce IT system testing cost

Orthogonal arrays are a powerful tool in quality and statistics. This paper shows new applications for improving testing effectiveness and efficiency in a multi-parameter environment, which is commonly encountered in today's software and systems. Unique aspects of this paper are applications in diverse areas — telecommunications, defense, automotive, information technology, and financial systems. Software and system testing cost the commercial and defense industry hundreds of millions of dollars annually. In addition, conducting each set of tests takes multiple man-months, delaying time to market of key technologies. In this current economic environment, organizations are looking for ways to reduce the cost of testing and time to market while ensuring that defects are not passed on to the customer. At the same time, the organizations are very reluctant to change their standard testing processes due to the heavy cost of field failures, regulatory concerns, and risk-averse culture. This paper describes a comparative study undertaken to assess the benefits of using Orthogonal Arrays (OA) for generating test plans in IT systems in the Financial Services industry. The formal process used for the comparative study consisted of enlisting the support of senior management and conducting multiple side-by-side pilots to compare the cost and risk of OA based testing versus the Business as Usual (BAU) testing practices. Our customers ran 20 side-by-side studies to evaluate the effectiveness of OA based testing and realized an average reduction in total test effort by 41%. In addition, all defects detected by the BAU process were detected by the OA based testing process. Further, in 40% of the cases, the OA based testing process found more defects and in the remaining 60% of the cases, BAU and Orthogonal Arrays found the same defects. The cost and schedule savings translated to tens of millions of dollars in labor and schedule.