H-GPR: A Hybrid Strategy for Large-Scale Gaussian Process Regression

With the massive volume of data emerging from both scientific and industrial domains, it has become a desideratum to improve the scalability of Gaussian process regression (GPR). There are two major approaches to assuage its $\mathcal{O}\left( {{n^3}} \right)$ training complexity: the aggregation based methods and the sparse approximation methods. This paper proposes a hybrid strategy called H-GPR to combine these two well-established approaches. We show that it is possible to improve the performance of aggregation based methods by removing some data points that severely violate its underlying assumption, and then this information loss can be recovered by a set of inducing points generated by the sparse approximation methods. A novel metric called conditional independent score is proposed, which measures to what extent the assumption made by the aggregation based methods is satisfied. A heuristic rule is developed to adjust the relative size of the local experts and the inducing subset, so that their distinction can be better reflected. Thorough experiments on synthetic and realistic datasets were performed, demonstrating that the proposed method can improve both the predictive means and variances.