Neural finite-state transducers: a bottom-up approach to natural language processing

Neural networks call for a formal model of computations that they carry out during solving natural language processing tasks. At present, formal symbolic models are a reference point for using neural networks. Such an approach may be called top-down because it assumes that neural computations are based on a direct or indirect manipulation of structured symbolic representations. In this paper we present a bottom-up approach, in which we do not make such an assumption. Starting from a direct interpretation of performance of binary recurrent neural networks during processing of symbol sequences, a formal model of transducer was introduced. An exemplary definition of such a transducer and its application to a part-of-speech tagging problem is illustrated. From its operation we conclude that neural computations are non-structural and based not on manipulating symbolic structures but on the principle of causality.