Comparison of Static Neural Network with External Memory and RNNs for Deterministic Context Free Language Learning

In this paper, a learning model for prediction is introduced by coupling a static neural network with an external stack memory, creating a new type of recurrent system. We analyze the differences between this external memory recurrent network and recurrent neural network, which possesses internal memory. Internal memory remembers the last state while external memory remembers past useful contents. For a specific automaton, the internal memory is needed if the last state is a variable of the state transition function and the external memory is needed if the past content is a variable of state transition function. Our arguments are verified by comparing the prediction accuracy of different models with internal memory, external memory and the combination of them on counting and reversing tasks. The results shows that: network with an external stack works best for counting tasks since the variables of state transition function is composed of the current input and one past input, while network with the combination of internal and external memory works best for the reversing task since the variables of state transition function is composed of the current input, current state and the past inputs.