The fractal geometry of backpropagation

It is empirically known that standard backpropagation is very sensitive to the initial learning rate chosen for a given learning task. In this paper the author examines the shape of the iteration path for the training of a linear associator using backpropagation with momentum and online backpropagation. In the first case the path resembles Lisajous figures, in the second it is a fractal in weight space. The specific form depends on the learning rate chosen, but there is a threshold value for which the attractor of the iteration path is dense in a region of weight space around a local minimum of the error function. This result also yields a deeper insight into the mechanics of the iteration process in the nonlinear case.<>