Efficient Hyper-Parameter Tuning for the Kappa Regression Algorithm

The Kappa Regression algorithm has been shown to provide great performance results, but hyper-parameter tuning can, at times, be computationally intensive. The algorithm in-cludes a single parameter for each feature and a global parameter is required when features are combined. For each feature, one can perform a grid (linear) search, but this can be inefficient if the range of parameter values is large. However, we have found, experimentally, that the prediction error is a convex function of the parameter value. Although we have not shown this analytically, we have demonstrated this property over a wide range of datasets. Using this property, we use a modified Golden Section Search algorithm to find the unique optimal parameter value. In order to initialize the algorithm, we need function values for two points such that the optimal parameter value lies somewhere in between. The left point (with a parameter value of zero) is easy to compute. However, the rightmost point of the algorithm is taken as twice the rough estimate of the optimal parameter value, which is computed using various statistical properties of the dataset. We demonstrate the efficiency of the approach using several datasets.