Extracting stochastic machines from recurrent neural networks trained on complex symbolic sequences

We train a recurrent neural network on a single, long, complex symbolic sequence with positive entropy. The training process is monitored through information theory based performance measures. We show that although the sequence is unpredictable, the network is able to code the sequence's topological and statistical structure in recurrent neuron activation scenarios. Such scenarios can be compactly represented through stochastic machines extracted from the trained network. Generative models, i.e. trained recurrent networks and extracted stochastic machines, are compared using entropy spectra of generated sequences. In addition, entropy spectra computed directly from the machines capture generalization abilities of extracted machines and are related to a machines' long term behavior.