Enhancing Sparsity of Support Vector Machines by Ridge Regression

Reducing the number of support vectors which construct the Support Vector Machine (SVM) has been a research topic of major significance. The paper introduces a framework consisting of three consecutive procedures to enhance sparsity of the SVM post its training. First, upon the observation that outliers always end up support vectors and are likely to compromise the generalization ability of the SVM, outliers are identified and removed from the training set after the SVM is trained. Next, assuming the normal vector can be spanned by a fixed number of basis functions, the paper propose to optimize the expansion coefficients by ridge regression. Meanwhile, a strategy is introduced to ensure linear independence among the basis functions. The resultant classifier is referred to as “Ridge Regression SVM” (RR-SVM). In the end, the support vectors are pruned iteratively from the RR-SVM. At each iteration, the LOOCV of the RR-SVM due to the absence of a support vector from the training set is computed efficiently. The support vector corresponding to the maximal LOOCV accuracy is eventually removed. Evaluation metrics alternative to the LOOCV accuracy are also developed and used for optimal pruning of the support vectors. Experiments on the checkerboard dataset and four datasets from the UCI repository confirm that, compared to the standard SVM, the RR-SVM maintains excellent generalization performance and features enhanced sparsity.