Kernel MSE algorithm: a unified framework for KFD, LS-SVM and KRR

We generalize the conventional minimum squared error (MSE) method to yield a new nonlinear learning machine by using the kernel idea and adding different regularization terms. We name it kernel minimum squared error (KMSE) algorithm, which can deal with linear and nonlinear classification and regression problems. With proper choices of the output coding schemes and regularization terms, we prove that KMSE is identical to the kernel Fisher discriminant (KFD) except for an unimportant scale factor, and it is directly equivalent to the least square version for support vector machine (LS-SVM). For continuous real output values, we find that KMSE is the kernel ridge regression (KRR) with a bias. Therefore KMSE can act as a general framework that includes KFD, LS-SVM and KRR as its particular cases. In addition, we simplify the formula to estimate the projecting direction of KFD. Experiments on artificial and real world data sets in numerical computation aspects demonstrate that KMSE is a class of powerful kernel learning machines.