Measuring AI Performance

Summary

Scoring in artificial intelligence competitions, medical applications, and all other “classification” problems use F1 score as it is the correct way to compare two different “classifiers,” but most retailers use accuracy, recall, or their own made up metric instead. This chapter highlights some examples of really bad ways to compare two out‐of‐stock detection scores: using accuracy instead of F1 score; measuring performance as recall only; measuring performance as precision only; and measuring ground truth and the vendor's predictions at different times. The chapter provides a good reference chart to know which performance metric the readers should use depending on what type of problem they are trying to solve: binary classification, multiclass classification, product recognition, and demand forecasting.