Interactive Evaluation of Classifiers Under Limited Resources

In this paper, we propose strategies to estimate the accuracy of classifiers on a dataset when resource limitations restrict the number of instances for which true labels can be obtained. Our target scenarios include situations where the classifier output labels, but no scores, e.g. when the "classifier" is not an automated classifier but an inexpert human labeller who only outputs labels. Our objective is to optimally select a subset of the data to obtain true labels for, such that they provide the best estimate of classifier accuracy. We use techniques based on stratified sampling to address this problem. However, stratified sampling poses two challenges: i) how best to stratify the data, and ii) how to allocate samples among the strata. We propose a method of stratifying data and then present two novel interactive algorithms to approximate optimal allocation of samples to the strata. Our proposed methods for stratification and allocation are seen to outperform other popular approaches to the problem.