Structure and parameter learning via entropy minimization, with applications to mixture and hidden Markov models

We develop a computationally efficient framework for finding compact and highly accurate hidden-variable models via entropy minimization. The main results are: (1) an entropic prior that favors small, unambiguous, maximally structured models. (2) A prior balancing manipulation of Bayes' rule that allows one to gradually introduce or remove constraints in the course of iterative reestimation. (1) and (2) combined give the information-theoretic free energy of the model and the means to manipulate it. (3) Maximum a posteriori (MAP) estimators such that entropy optimization and deterministic annealing can be performed wholly within expectation maximization (EM). (4) Trimming tests that identify excess parameters whose removal will increase the posterior, thereby simplifying the model and preventing over-fitting. The end result is a fast and exact hill-climbing algorithm that mixes continuous and combinatoric optimization and evades sub-optimal equilibria. Examples are given using speech and language problems.