Bayesian inference for LS-SVMs on large data sets using the Nystrom method

In support vector machines (SVMs), least squares SVMs (LS-SVMs) and other kernel based techniques for regression and classification the solution follows from a convex optimization problem for a fixed choice of the hyper-parameters. However, these methods involve the calculation, storage and typically also inversion of the kernel matrix with the size equal to the number of data points. Therefore, large scale techniques like sequential minimal optimization and conjugate gradient algorithms have been developed in order to train the SVM and LS-SVM on large data sets, respectively. In Bayesian inference for SVMs and LS-SVMs one also needs to compute the inverse and eigenvalue decomposition of the kernel matrix, which is again computationally intensive. In this paper, we discuss large scale approximations for Bayesian inference for LS-SVMs. A practical implementation using the Nystrom method is developed which allows one to obtain approximate expressions at the different levels of inference within the evidence framework. The method is then evaluated on a number of benchmark problems.