A new multidimensional penalized likelihood regression method

Penalized likelihood regression is a concept whereby the log-likelihood of the observations is combined with a term measuring the smoothness of the fit, and the resulting expression is then optimized. This concept vies for achieving a compromise between goodness of fit (as typified by the likelihood function) and smoothness of the data. Penalized likelihood regression, which has been developed in the statistics literature since the seventies, has focused mostly on the one-dimensional case. Attempts to consider the general multidimensional case have been limited. In this paper we propose a new multidimensional penalized likelihood regression method. The approach is based on proposing a roughness term based on the discrepancy between the function values among the K-nearest-neighbors. The proposed formulation yields a simple solution in terms of a system of linear equations. We also derive an iterative solution to the problem that sheds light on its basic functionality. The iteration consists of repeatedly taking the weighted average of the target output value and the estimated function values of the K-nearest-neighbors. We show that the proposed model is fairly versatile in that it exhibits nice features in handling user-defined function constraints and data imperfections. Experimental results confirm that it is competitive with the Gaussian process regression method (one of the best methods out there), and exhibits significant speed advantage.