Entropy Estimation from Queries and Auxiliary Distribution Samples

We consider the problem of estimating the Shannon entropy of an unknown probability mass function (PMF) $P$ from a countable alphabet with a finite unknown support. The estimator may query the exact value of $P$ in any specific letter, but this operation is costly, and hence the amount of queries should be minimized. In parallel, the estimator can sample from an auxiliary PMF $Q$, which is assumed to be close to $P$ in some sense. We propose and analyze two specific algorithms for this setting. The first is based on importance sampling and is shown to be asymptotically unbiased and consistent, but does not limit the number of queries. The second algorithm, called $Q$-sort, limits the number of queries. We prove that it has an exponentially decaying MSE. Finally, we also provide a simple characterization minimax lower bound on the error rate, which also tends to zero exponentially fast with the number of samples.