Probabilistic memory capacity of recurrent neural networks

In this paper, probabilistic memory capacity of recurrent neural networks (RNNs) is investigated. This probabilistic capacity is determined uniquely if the network architecture and the number of patterns to be memorized are fixed. It is independent from the learning method and network dynamics. It provides the upper bound of the memory capacity by any learning algorithm in memorizing random patterns. It is assumed that the network consists of N units which take two states. Thus, the total number of patterns is the Nth power of 2. The probabilities are obtained by discrimination whether the connection weights, which can store random M patterns at equilibrium states, exist or not. A theoretical approach for this purpose is derived, and the actual calculation is executed by the Monte Carlo method. As an example of the learning algorithm, an improved error correction learning is investigated, and its convergence probabilities are compared with the upper bound. A linear programming method can be effectively applied to this numerical analysis.