Relevance and insight in experimental studies

As its name suggests, artificial intelligence is a science of the artificial. As with other conscious creations, there is a great temptation to assume that we can understand the behavior of AI systems entirely through formal analysis. However, the complexity of most AI constructs makes this impractical, forcing us to rely on the same experimental approach that has been so useful in the natural sciences. Many of the same issues and methods apply directly to AI systems, including the need to identify clearly one's dependent and independent variables, the importance of careful experimental design, and the need to average across random variables outside one's control. However, beyond these obvious features, a compelling experimental study of intelligent behavior must satisfy two additional criteria: it must have relevance and it must produce insight. I will illustrate these ideas with examples from machine learning, one of the most experimentally oriented subfields within artificial intelligence. Moreover, because AI researchers are often concerned with extending some existing method to improve its behavior, I will focus on this paradigm.