Shocking: an approach to stabilize backprop training with greedy adaptive learning rates

In general, backpropagation neural networks converge faster with adaptive learning rates than with learning rates that remain constant or grow or decay disregarding to the network error. This is because each synapse has its own learning rate that can vary over time by an amount appropriate to that weight. However, adaptive learning rates cause neural networks to saturate during training. When learning rates are permitted to assume values greater than unity, they are considered "greedy". Greedy adaptive learning rates can reduce not only the training times of networks, but also compromise the stability of the training process, leading to a network that fails to converge. This paper proposes a simple ad hoc approach called "shocking" as a partial solution to the instability problem caused by greedy adaptive learning rates. An analysis based on training times and failure rates for two inherently unstable benchmark problems is used to validate the use of shocking.