Information Gain and Uniform Generalization Bounds for Neural Kernel Models

Neural tangent (NT) kernel models have attracted great attention lately, partly for explaining overparameterized neural networks in their limit. In previous work, generalization bounds for overparameterized neural networks were given in expectation. In this work, we prove uniform generalization bounds for NT kernel models by characterizing the information gain of the NT kernel. Our bounds capture the exact error rates based on the smoothness of the activation functions. Our results on information gain can be applied to other problems where overparameterized neural networks are used; e.g., certain reinforcement learning and bandit problems.