Getting what you deserve from data

This article's focus appears at first to be a narrow, prescriptive little corner of the methodological landscape. Data analysis is often dismissed as no more complicated than calculating some means and comparing them with t tests or the like. Consequently, experiments and analyses are inefficient, requiring more data than necessary to show an effect; they waste data, failing to show effects; and they sometimes induce hallucinations, suggesting effects that don't exist. Bad analysis can spoil an entire research program, so it warrants attention. I will discuss three common and easily fixed problems: accepting the null hypothesis, a misuse of statistical machinery; inadequate attention to sources of variance, leading to insignificant results and failure to notice interactions among factors; and multiple pairwise comparisons, leading to nonexistent effects.