Measures for ranking estimation performance based on single or multiple performance metrics

There are several error metrics for estimation performance evaluation. To rank the performance of estimators, a popular method is using the same error metric of performance. It is not without controversy. First, this ranking method depending on the “marginal” information without considering the “joint” information among the estimators is one-sided since different error metrics reflect different aspects of performance. Second, ranking according to different error metrics may lead to different results. Thus, we propose to use the “joint” information just like Pitman's closeness measure (PCM) to rank the performance of estimators. However, one drawback of PCM, named nontransitivity, brings big trouble for estimation performance ranking. To rank estimators utilizing the “joint” information, we propose a new approach using a so-called estimator ranking vector (ERV). The elements of ERV reflect how good the corresponding estimators are. Order-preserving mappings are proposed to obtain ERV, which, however, may not be unique. Then we use three specific mappings (i.e., linear, contraction, and concave, respectively) to solve this problem. Linear mappings can be easily applied and the other two mappings broaden the application domain of ERV. The ranking vector can also be used in multiple-attribute ranking problem. It does not need data normalization.