On accuracy estimation and comparison of results in biometric research

The estimated accuracy of an algorithm is the most important element of the typical biometrics research publication. Comparisons between algorithms are commonly made based on estimated accuracies reported in different publications. However, even when the same dataset is used in two publications, there is a very low frequency of the publications using the same protocol for estimating algorithm accuracy. Using the example problems of face recognition, expression recognition and gender classification, we show that the variation in estimated performance on the same dataset across different protocols can be enormous. Based on these results, we make recommendations for how to obtain performance estimates that allow reliable comparison between algorithms.