Statistical Evaluation of Complex Input-Output Transformations

This paper presents a new, statistical approach to evaluating software products that transform complex inputs into complex outputs. This approach, called multistage stratified input/output (MSIO) sampling, combines automatic clustering of multidimensional I/O data with multistage sampling and manual examination of data elements, in order to accurately and economically estimate summary measures of output data quality. We report results of two case studies in which MSIO sampling was successfully applied to evaluating complex graphical outputs.