Measuring and assessing software test processes using test data

Testing of large-scale software systems is a complex and expensive process that involves both technical and managerial issues. To improve its cost-effectiveness, the process should be continuously monitored, consistently measured and carefully assessed. This paper proposes an assessment methodology in this direction, called process-oriented metrics-based assessment (POMBA). Novel concepts include considering test problems as one of the dependent variables of the test process and the use of test intensity as one of the independent variables. As a proof-of-concept, the methodology is applied to test data collected from high-level Y2K (Year-2000) tests of seven large-scale transaction software systems. It is found that, in testing large-scale software, test problems that prevent the completion of tests due to insufficient test planning and setup activities tend to occur frequently. This not only wastes resources but also affects the effectiveness of the overall process.