A Spiking Neural Network Learning Markov Chain

In this paper, the question how spiking neural network (SNN) learns and fixes in its internal structures a model of external world dynamics is explored. This question is important for SNN implementation of the model-based reinforcement learning (RL), finding anomalies in time series and other problems. In the present work, we formalize world dynamics as a Markov chain with a priori unknown state transition probabilities, which should be learnt by the network. To make this problem formulation more realistic, we solve it in the continuous time, so that the duration of every state in the Markov chain may be different and is unknown. It is demonstrated how this task can be accomplished by an SNN with a specially designed structure and local synaptic plasticity rules. As an example, we show how this network motif works in the simple but non-trivial world where a ball moves inside a square box and bounces from its walls with a random new direction and velocity.