Design of Experiments in Computer Performance Evaluation

Techniques of statistical design of experiments have been successfully employed for many decades in a variety of applications in industry, agriculture, medicine, psychology, and other physical and social sciences. Their aim is to provide scientific and efficient means of studying the effects, on one or more variables of interest, of varying multiple controllable factors in an experiment. These techniques have not been widely used in the study of computer systems, although they can potentially have as large an impact as they have had in other fields. The purpose of this paper is to review some of the basic concepts underlying the statistical design and analysis of experiments and to illustrate them by means of examples drawn from studies of computer system performance. The examples include comparisons of alternate page replacement and free storage management algorithms, optimization of a scheduler, and validation of a system simulation model.