A Family of Hyperparameter Estimators Linking EB and SURE for Kernel-Based Regularization Methods

Hyperparameter estimation is a critical aspect of kernel-based regularization methods (KRMs), alongside kernel design. Empirical Bayes (EB) and Stein's unbiased risk estimator (SURE) are two widely used hyperparameter estimators for tuning the unknown hyperparameters associated with the kernel matrix of KRMs. However, EB and SURE exhibit different characteristics in both theory and practice. Theoretically, SURE is asymptotically optimal in terms of minimizing the mean squared error (MSE), whereas EB generally is not. However, practical evidence suggests that EB is often more accurate and robust than SURE, especially when the regression matrix is ill-conditioned. Therefore, this article aims to deepen our understanding of these two estimators in a unified manner. First, we construct a family of hyperparameter estimators that encompasses both EB and SURE by introducing an index. We then explain the loss function of this family as a tradeoff between data fit and model complexity for regularized least squares estimators, using least squares estimators as a reference. Second, we establish the convergence and rate of convergence of this family and further demonstrate that it is asymptotically optimal in some weighted MSE for a specific kernel matrix. Finally, Monte-Carlo simulations indicate the existence of other hyperparameter estimators in this family that outperform both EB and SURE methods for certain datasets.