Sparse Gaussian Process Regression using Progressively Growing Learning Representations

We present a new sparse Gaussian process regression model whose covariance function is parameterized by the locations of a progressively growing set of pseudo-inputs generated by an online deterministic annealing optimization algorithm. A series of entropy-regularized optimization problems is solved sequentially, introducing a bifurcation phenomenon, according to which, pseudo-inputs are gradually generated. This results in an active learning approach, which, in contrast to most existing works, can modify already selected pseudo-inputs and is trained using a recursive gradient-free stochastic approximation algorithm. Finally, the proposed algorithm is able to incorporate prior knowledge in the form of a probability density, according to which new observations are sampled. Experimental results showcase the efficacy and potential advantages of the proposed methodology.