Optimal Convergence for Agnostic Kernel Learning With Random Features

Owing to their solid theoretical guarantees and flexible learning framework, random features (RFs) methods have drawn increasing attention in the field of nonparametric statistical learning. However, existing studies on RFs assume that the target function lies exactly in the associated kernel space, which may not hold true in practical applications. In this article, we investigate the effectiveness of RFs in an agnostic setting that the target regression may be out of the kernel space and prove that they can still achieve capacity-dependent statistical optimality. To achieve this, we provide a finer grained estimate for the capacity of the hypothesis space, and conduct a refined analysis of error terms after a concise error decomposition. Our results show that RF with uniform sampling can guarantee optimality in half of the agnostic situations, while RF with data-dependent sampling can achieve optimal rates in the entire agnostic setting. This finding suggests that using data-dependent sampling not only reduces the number of RFs but also improves their applicability in agnostic settings. Finally, we compare the performance of RFs with different sampling strategies on several real-world datasets. The experimental results provide supports for our theoretical findings.