Empirical divergence maximization for quantizer design: An analysis of approximation error

Empirical divergence maximization is an estimation method similar to empirical risk minimization whereby the Kullback-Leibler divergence is maximized over a class of functions that induce probability distributions. We use this method as a design strategy for quantizers whose output will ultimately be used to make a decision about the quantizer's input. We derive this estimator's approximation er ror decay rate as a function of the resolution of a class of partitions known as recursive dyadic partitions. This result, coupled with ear lier results, show that this estimator can converge to the theoretically optimal solution as fast as n
^-1
, where n is the number of training samples. This estimator also is capable of producing estimates that well-approximate optimal solutions that existing techniques cannot.