Error Estimation for Discrete Classification

The study of error estimation for discrete classifiers is a fertile topic, as analytical characterizations of performance are often possible due to the simplicity of the problem. This chapter provides the definitions and simple properties of the main error estimators for the discrete histogram rule, which is the most important example of a discrete classification rule. The error estimators discussed in the chapter include resubstitution error, leave-one-out error, cross-validation error, and bootstrap error estimator. A detailed analytical study of small-sample performance in terms of bias, deviation variance, RMS, and correlation coefficient between true and estimated errors is presented. The chapter illustrates the exact bias, deviation standard deviation, and RMS of resubstitution and leave-one-out, plotted as functions of the number of bins. For comparison, Monte-Carlo estimates of 10 repetitions of 4-fold cross-validation are also plotted. The chapter also presents a complete enumeration approach and analyses large-sample performance.