Nearly Consistent Finite Particle Estimates in Streaming Importance Sampling

In Bayesian inference, we seek to compute information about random variables such as moments or quantiles on the basis of available data and prior information. When the distribution of those random variables is intractable, Monte Carlo (MC) sampling is usually required. Importance sampling is a standard MC tool that approximates this unavailable distribution with a set of weighted samples. This procedure is asymptotically consistent as the number of MC samples (particles) goes to infinity. However, retaining infinitely many particles is infeasible. Thus, we propose a way to only keep a
finite representative subset
of particles and their augmented importance weights that is
nearly consistent
, i.e., they converge to a close neighborhood of their population ground truth in the large sample limit. To do so in an online manner, we (1) embed the posterior density estimate in a reproducing kernel Hilbert space (RKHS) through its kernel mean embedding; and (2) sequentially project this RKHS element onto a lower-dimensional subspace in RKHS using the maximum mean discrepancy, an integral probability metric. Theoretically, we establish that this scheme results in a bias determined by a compression parameter, which yields a tunable tradeoff between consistency and memory. In experiments, we observe the compressed estimates achieve comparable performance to the dense ones with substantial reductions in representational complexity.