Exploring the Global Dynamics of Networks Trained through Equilibrium Propagation

Equilibrium propagation is a learning technique conceived for training continuous-time recurrent neural networks. It offers some notable advantages when compared to conventional back-propagation-based algorithms and to classical design methods. From an implementation perspective, it demands only a single computational circuit. Theoretically, although it seeks to minimize a cost function, it exhibits similarities to spike-timing-dependent plasticity (STDP), rendering it, to a certain extent, biologically plausible. This paper explores the global dynamic behavior of continuous-time piecewise linear networks trained through equilibrium point propagation. We examine a network in which the target patterns are presented as external inputs rather than as initial conditions. We first show that the learning rules, which extend equilibrium propagation to gradient-like and non-symmetric networks, can be derived as a suitable approximation of Lagrangian optimization. Then, by studying a relatively simple but thoroughly significant case, we demonstrate that a detailed analysis of the equilibrium point distribution yields a profound understanding of the network’s fundamental properties and provides a valuable tool for quantitatively evaluating the network’s accuracy. Compared to classical synthesis techniques, our approach, where patterns are introduced as external inputs, in most cases, circumvents the impractical task of estimating the basins of attraction for sets of multiple equilibrium points. Furthermore, preliminary extensive simulations indicate that the primary dynamic features observed in relatively small networks closely resemble those ensuring the performance and accuracy of large-scale networks.