The Kernel Conjugate Gradient Algorithms

Kernel methods have been successfully applied to nonlinear problems in machine learning and signal processing. Various kernel-based algorithms have been proposed over the last two decades. In this paper, we investigate the kernel conjugate gradient (KCG) algorithms in both batch and online modes. By expressing the solution vector of CG algorithm as a linear combination of the input vectors and using the kernel trick, we developed the KCG algorithm for batch mode. Because the CG algorithm is iterative in nature, it can greatly reduce the computations by the technique of reduced-rank processing. Moreover, the reduced-rank processing can provide the robustness against the problem of overlearning. The online KCG algorithm is also derived, which converges as fast as the kernel recursive least squares (KRLS) algorithm, but the computational cost is only a quarter of that of the KRLS algorithm. Another attractive feature of the online KCG algorithm compared with other kernel adaptive algorithms is that it does not require the user-defined parameters. To control the growth of data size in online applications, a simple sparsification criterion based on the angles among elements in reproducing kernel Hilbert space is proposed. The angle criterion is equivalent to the coherence criterion but does not require the kernel to be unit norm. Finally, numerical experiments are provided to illustrate the effectiveness of the proposed algorithms.