Unbiased support vector classifiers

Support Vector Classifiers (SVC) are claimed to provide a natural mechanism for implementing Structural Risk Minimization (SRM), obtaining machines with good generalization capabilities. SVC leads to the optimal hyperplane (maximal margin) criterion for separable datasets but, in the nonseparable case, a functional with an additional term has to be minimized. The particular form of this extra term is such that the minimization can be solved via Quadratic Programming (QP), but, in this case, it represents a rather coarse approximation to the number of errors. We propose an unbiased implementation of SVC by introducing a more appropriate "error counting" term. This way, the number of classification errors is truly minimized (hence the "unbiased" appellative), while the maximal margin solution is obtained in the separable case. QP can no longer be used for solving the new minimization problem, and we apply instead an iterated Weighted Least Squares (WLS) procedure. Computer experiments show that the proposed method is superior to the classical approach in terms of both classification error and machine complexity.