MLP emulation of N-gram models as a first step to connectionist language modeling

In problems such as automatic speech recognition and machine translation, where the system response must be a sentence in a given language, language models are employed in order to improve system performance. These language models are usually N-gram models (for instance, bigram or trigram models) which are estimated from large text databases using the occurrence frequencies of these N-grams. Nakamura and Shikano (1989) empirically showed how multilayer perceptrons can emulate trigram model predictive capabilities with additional generalization features. Our paper discusses Nakamura and Shikano's work, provides new empirical evidence on multilayer perceptron capability to emulate N-gram models, and proposes new directions for extending neural network-based language models. The experimental work we present here compares connectionist phonological bigram models with a conventional one using different measures, which include recognition performances in a Spanish acoustic-phonetic decoding task.