Markov random field models for natural language

Markov chain (N-gram) source models for natural language were explored by Shannon and have found wide application in speech recognition systems. However, the underlying linear graph structure is inadequate to express the hierarchical structure of language necessary for encoding syntactic information. Context-free language models which generate tree graphs are a natural way of encoding this information, but lack the modeling of interword dependencies. We consider a hybrid tree/chain graph structure which has the advantage of incorporating lexical dependencies in syntactic representations. Two Markov random field probability measures are derived on these tree/chain graphs from the maximum entropy principle.