Taking the mystery out of experimental design-and a proposal

Summary form only given. As the computer science and software engineering community recognizes the importance of empirical studies, we look to many other disciplines for models of empirical work. At the core of many of these models are methods and techniques refined over the years that are known to be credible to the discipline's typical environments. One way of capturing this knowledge is to abstract the method into an experimental design, but what is an experimental design? I try to take some mystery out of experimental design. The key ideas are to recognize that science and engineering are founded upon credibility, which can range in empirical studies from the weak credibility of specific, non-verifiable, anecdotal reports to the strong credibility of experiments to show a casual relationship with that of the case study somewhere in the middle. The experimental design is the logic and interpretation infrastructure that attempts to show a causal relationship. I describe several experimental designs from different disciplines and abstract the key elements to form an operational definition. Finally, I observe that computer science and software engineering environments are similar to the biological sciences and conclude that an excellent match for our environments is the in vivo and in vitro empirical model and their experimental designs.