The scientific method and design and test

A few years ago, during a well-attended open Design Automation Conference benchmark forum, a panellist pointed out that "...reporting experimental results is a science and an art. A survey of the literature may reveal a consistent methodology.... Also, we should address the verification of reported results." In retrospect, the suggestion may have hinted at the vast body of techniques commonly known as experimental design. However, this forum was not ready to expand on the subject. Most search engines currently on the Web return tens of thousands of URLs in response to keyword searches using terms such as experimental design or design of experiments. Few, if any, of these search results point to an evaluation of CAD algorithms. In contrast, the experimental design methodology, pioneered by Fischer during the 1920s in agricultural research, is now firmly established in science and manufacturing. Its application to biomedical research can save lives. Biomedical journals have strict guidelines on how to report experimental results so others may replicate experiments. A URL from a medical school points to a concise illustration of a simple experimental design flow and the terminology used. Adopting the accepted norms of experimental design will give us a scientific method to conduct, verify, and report comparative performance evaluations of CAD algorithms.