Robustness of comparison sequential test for the piloting in Service Delivery

This paper discusses a piloting methodology that supports a complicated decision-making process of evaluating an Information Technology process or function relative to the business value it generates. We use sequential comparison testing as a direct tool for performance evaluation of the operational innovation in volatile Service Delivery processes. Sequential tests are exceptionally suitable for finding the best process or function, and also for statistically significant measurement of the process parameters. Since the characteristics of a service delivery process vary widely, on the one hand that evaluation should be comparative, i.e. matched against that of another group working in parallel and serving as the reference. On the other hand, there is a question of the correctness of these measurements under changing parameters, i.e. of the robustness of piloting. We studied robustness of the proposed sequential comparison tests in the context in question and demonstrated that it only depends on the coefficients of variation of the compared processes and not on the other parameters of their distributions. We established that the test is indeed robust in the settings of IT Service delivery environment. We also show that it is superior to the alternative fixed sample size test and pairwise sequential test. This study provided a statistically rigorous background for application of proposed test to pilot based measurements.