Deep Kernel Learning Networks with Multiple Learning Paths

This paper proposes deep kernel learning networks with multiple learning paths (DKL-MLP) for nonlinear function approximation. Leveraging the random feature (RF) mapping technique, kernel methods can be implemented as a two-layer neural network, at drastically reduced workload on weight training. Motivated by the representation power of the deep architecture in deep neural networks, we devise a vanilla deep kernel learning network (DKL) by applying RF mapping at each layer and learn the last layer only. To improve the learning performance of DKL, we add multiple trainable paths to DKL and develop the DKL-MLP method so that some implicit information from earlier hidden layers to the output layer can be learned. We prove that both DKL and DKL-MLP permit universal representation of a wide variety of interesting functions with arbitrarily small error and have no bad local minimum. Numerical experiments on both regression and classification tasks are provided to demonstrate the learning performance and computational efficiency of the proposed methods.