A Family of Hyper-parameter Estimators Linking EB and SURE for Kernel-based Regularization Methods

Hyper-parameter estimation is a critical aspect of kernel-based regularization methods (KRMs), alongside kernel design. Empirical Bayes (EB) and Stein's unbiased risk estimator (SURE) are two widely used hyper-parameter estimators for tuning the unknown hyper-parameters associated with the kernel matrix of KRMs. However, EB and SURE exhibit different characteristics in both theory and practice. Theoretically, SURE is asymptotically optimal in terms of minimizing the mean squared error (MSE), whereas EB generally is not. However, practical evidence suggests that EB is often more accurate and robust than SURE, especially when the regression matrix is ill-conditioned. Therefore, this paper aims to deepen our understanding of these two estimators in a unified manner. Firstly, we construct a family of hyper-parameter estimators that encompasses both EB and SURE by introducing an index. We then explain the loss function of this family as a trade-off between data fit and model complexity for regularized least squares estimators, using least squares estimators as a reference. Secondly, we establish the convergence and rate of convergence of this family and further demonstrate that it is asymptotically optimal in some weighted MSE for a specific kernel matrix. Finally, Monte-Carlo simulations indicate the existence of other hyper-parameter estimators in this family that outperform both EB and SURE methods for certain datasets.