Loading [MathJax]/extensions/MathMenu.js
Error Surface of Recurrent Neural Networks | IEEE Journals & Magazine | IEEE Xplore

Error Surface of Recurrent Neural Networks


Abstract:

We found in previous work that the error surfaces of recurrent networks have spurious valleys that can cause significant difficulties in training these networks. Our earl...Show More

Abstract:

We found in previous work that the error surfaces of recurrent networks have spurious valleys that can cause significant difficulties in training these networks. Our earlier work focused on single-layer networks. In this paper, we extend the previous results to general layered digital dynamic networks. We describe two types of spurious valleys that appear in the error surfaces of these networks. These valleys are not affected by the desired network output (or by the problem that the network is trying to solve). They depend only on the input sequence and the architecture of the network. The insights gained from this analysis suggest procedures for improving the training of recurrent neural networks.
Published in: IEEE Transactions on Neural Networks and Learning Systems ( Volume: 24, Issue: 11, November 2013)
Page(s): 1709 - 1721
Date of Publication: 25 June 2013

ISSN Information:

PubMed ID: 24808606

Contact IEEE to Subscribe

References

References is not available for this document.