Statistical tools for evaluating the adequacy of the size of a sample on which statistical evidence is based

Although courts have incorporated statistical hypothesis testing into their evaluation of numerical evidence in a variety of cases, they have primarily focused on one aspect of a statistical analysis: whether or not the result is ‘statistically significant’ at the 0.05 or ‘two-standard deviation’ level. The theory underlying hypothesis testing is also concerned with the power of the test to detect a meaningful difference. This article shows that using the insights provided by power calculations should assist courts to better interpret and evaluate the statistical analyses submitted into evidence. In particular, the concept of power should help in assessing whether a sample is too small to provide reliable inferences. On the other hand very large samples can classify minor differences as statistically significant. This occurs when the power of the test at the standard 0.05 level is very high. It will be seen that requiring significance at a more stringent level, e.g. 0.005, which can be determined from a power calculation, often resolves this problem.