A Dynamical Systems Perspective on Online Bayesian Nonparametric Estimators with Adaptive Hyperparameters

This paper presents and analyzes constant step size stochastic gradient algorithms in reproducing kernel Hilbert Space (RKHS), which encapsulates various adaptive nonlinear interpolation schemes. The hyperparameters of the function iterates are updated via a distribution that depends on the estimates generated by the algorithm. Using stochastic averaging theory, we show that the estimates generated by the algorithm converge weakly to an algebraically constrained ordinary differential equation. We illustrate this proposed algorithm in an online multi-class classification problem. Specifically, the proposed RKHS-valued stochastic gradient algorithm operating in concert with a Gaussian kernel whose bandwidth stably evolves during training, performs comparably to when the bandwidth is set according to oracle knowledge of its optimal value.