Evaluation of Estimation Algorithms: Credibility Tests

Assessments of estimation performance are often available. For example, many statistical estimators and filters provide assessments of the first two moments of their own estimation error (i.e., mean-square error [MSE] or error covariance matrix and bias). Are these assessments credible in that they reflect the true situation? The paper addresses this important yet little studied topic, referred to as the credibility of the assessments (or the estimators that make the assessments). We define the concept of credibility and formulate three classes of commonly encountered credibility-testing problems: MSE alone, bias alone, and MSE and bias jointly. Taking advantage of results in multivariate statistical analysis, we present several statistical tests for the credibility problems formulated and analyze and discuss in detail pros and cons of the proposed tests, contrasting with the existing test. How these tests can be used and how they perform are illustrated by representative numerical examples. For the existing MSE credibility test, we explain its underlying principle and analyze, discuss, and demonstrate its drawbacks and limitations. We also propose a test for comparing different credibility assessments.