Back-propagation heuristics: a study of the extended delta-bar-delta algorithm

An investigation is presented of an extension, proposed by A.A. Minai and R.D. Williams (Proc. Int. Joint Conf. on Neural Networks, vol.1, p.676-79, Washington, DC, 1990), to an algorithm for training neural networks in real-valued, continuous approximation domains. Specifically, the most effective aspects of the proposed extension are isolated. It is found that while momentum is particularly useful for the delta-bar-delta algorithm, it cannot be used conveniently because of sensitivity considerations. It is also demonstrated that by using more subtle versions of the algorithm, the advantages of momentum can be retained without any significant drawbacks