Minimum discrimination information clustering: modeling and quantization with Gauss mixtures

Gauss mixtures have gained popularity in statistics and statistical signal processing applications for a variety of reasons, including their ability to approximate well a large class of interesting densities and the availability of algorithms such as EM for constructing the models based on observed data. We here consider a different motivation and framework based on the information theoretic view of Gaussian sources as a "worst case" for compression developed by D.J. Sakrison (see IEEE Trans. Inform. Theory, vol.21, p.301-9, 1975) and A. Lapidoth (see IEEE Trans. Inform. Theory, vol.43, p.38-47, 1997). This provides an approach for clustering Gauss mixture models using a minimum discrimination distortion measure and provides the intuitive support that good modeling is equivalent to good compression. A simple example of a clustered Gauss mixture model applied to image archiving and querying is presented and and compared with the common color histogram method. Signatures for both query and target images were formed by encoding an image using the minimum distortion encoder to obtain a histogram for the components. A simple decision tree was designed to decide whether or not a "match" occurred between the query image (representing its type) and the target image based on the component histogram of each.