Iterative k Data Algorithm for solving both the least squares SVM and the system of linear equations

We introduce a novel learning algorithm dubbed Iterative k Data Algorithm (IkDA) for solving a system of linear equations having symmetric positive definite matrix (SPD) when direct solution is not feasible. More specifically, we apply it to both a system of linear equations and to the least squares support vector machines (LS SVM). The new algorithm is an extension of the Iterative Single Data Algorithm (ISDA) which is an excellent, coordinate descent, approach for training SVMs. ISDA performs an optimization along a single variable which is, in fact, a Gauss-Seidel method. Unlike the former, IkDA searches for a minimum of an SVM's quadratic cost function over the subspace of k worst violating data i.e. coordinates. The novel algorithm shows a superior performance in respect to ISDA and consequently to all the other SVM training approaches slower than ISDA. Hence, IkDA is very promising for classifying large and ultra-large datasets when direct solution of LS SVM model is not feasible.