The Y-Test: Fairly Comparing Experimental Setups with Unequal Effort

Evolutionary Computation has been dogged by a central statistical issue: how does one fairly compare the performance of two techniques which differ in the amount of work required? While Koza's computational effort statistic attempts to answer this problem, it is a point statistic and has other statistical problems. We present the j-test, a statistical test which takes as input a set of outcomes from the observed runs of two processes A and B. The j-test synthetically performs a work-balanced comparison between k runs of A and / runs of B. We show that by choosing k and / appropriately, we can compensate for the fact that one of the processes is computationally more efficient than the other.