Consistent Online Gaussian Process Regression Without the Sample Complexity Bottleneck

Gaussian process regression provides a framework for nonlinear nonparametric Bayesian inference applicable across machine learning, robotics, chemical engineering, and other settings. Unfortunately, the computational burden of the posterior mean and covariance scales cubically with the training sample size. Even worse, in the online setting where samples perpetually arrive, this complexity approaches infinity. Thus, popular perception is that Gaussian processes cannot be used with streaming data, and that approximations are required. Motivated by this necessity, we develop the first compression sub-routine for online Gaussian processes that preserves their convergence to the population posterior, i.e., asymptotic posterior consistency, while ameliorating their intractable complexity growth with the sample size. We do so by after each sequential Bayesian update, fixing an error neighborhood with respect to the Hellinger metric centered at the current empirical probability measure, and greedily tossing out past kernel dictionary elements until we hit the boundary of this neighborhood. We call the resulting method Parsimonious Online Gaussian Processes (POG). When we set the error radius, or compression budget, go to null with the sample size, then exact asymptotic consistency is preserved (Theorem li) at the cost of unbounded memory in the limit. On the other hand, for constant compression budget, POG converges to a neighborhood of the population posterior distribution (Theorem 1ii) but with finite memory that is at-worst determined by the metric entropy of the feature space (Theorem 2). Experiments on benchmark data demonstrates that POG exhibits favorable performance in practice.