A bootstrap interval estimator for Bayes' classification error

Using a finite-length training set, we propose a new estimation approach suitable as an interval estimate of the Bayes-optimal classification error L*. We arrive at this estimate by constructing bootstrap training sets of varying size from the fixed, finite-length original training set. We assume a power-law decay curve for the unconditional error rate as a function of training sample size n, and fit the bootstrap estimated unconditional error rate curve to this power-law form. Using a result from Devijver, we do this twice, once for the k nearest neighbor (kNN) rule to provide an upper bound on L* and again for Hellman's (k; k') nearest neighbor rule with reject option, which gives a lower bound for L*. The result is an asymptotic interval estimate of L* from a finite-length training sample. We apply our estimator to two classification examples, obtaining Bayes' error estimates.