Fixed-weight on-line learning

Conventional neural nets perform functional mappings from their input space to their output space in a manner analogous to long-term biological memory. This paper presents a method of designing neural nets where recurrent signal loops store knowledge in a manner analogous to short-term memory. The synaptic weights encode a learning algorithm. This gives these nets the ability to dynamically learn any functional mapping from a (possibly very large) set, without changing any synaptic weight. These nets are adaptive dynamic systems. Learning is online continually taking place as part of the net's overall behavior instead of a separate, externally driven process. We present four high-order fixed-weight learning nets. Two of these have standard backpropagation embedded in their synaptic weights. The other two utilize a more efficient gradient-descent-based rule. This scheme was discovered by examining variations in fixed-weight topology. We present empirical tests showing that all these nets could successfully learn functions from both discrete (Boolean) and continuous function sets. Largely, the networks were robust with respect to perturbations in the synaptic weights. The exception was the recurrent connections used to store information. These required a tight tolerance of 0.5%. The cost of these nets scaled approximately in proportion to the total number of synapses. We consider evolving fixed weight networks tailored to a specific problem class by analyzing the meta-learning cost surface of the networks presented.