Comprehensive evaluation of decision performance

Decision performance metrics abound, but they all have three fundamental limitations: (a) none are unified for diverse situations including, in particular, the case with mis-specified truth; (b) none are comprehensive; and (c) they all rely on perfect knowledge of the ground truth. In this paper, we present a unified metric that measures decision performance comprehensively without the need to know the ground truth. It is truly a distance metric and enjoys many nice properties.