By Topic

Preprocessing based solution for the vanishing gradient problem in recurrent neural networks

Sign In

Cookies must be enabled to login.After enabling cookies , please use refresh or reload or ctrl+f5 on the browser for the login options.

Formats Non-Member Member
$33 $13
Learn how you can qualify for the best price for this item!
Become an IEEE Member or Subscribe to
IEEE Xplore for exclusive pricing!
close button

puzzle piece

IEEE membership options for an individual and IEEE Xplore subscriptions for an organization offer the most affordable access to essential journal articles, conference papers, standards, eBooks, and eLearning courses.

Learn more about:

IEEE membership

IEEE Xplore subscriptions

3 Author(s)
Squartini, S. ; Dipt. di Elettronica e Autom., Ancona Univ., Italy ; Hussain, A. ; Piazza, F.

In this paper, a possible solution to the vanishing gradient problem in recurrent neural networks (RNN) is proposed. The main idea consists of pre-processing the signal (a time series typically) through a wavelet decomposition, in order to separate the short term information from the long term one, and treating each scale by different RNNs. The partial results concerning all the different scales of time and frequencies are combined by another 'expert' (a nonlinear structure typically) in order to achieve the final goal. This new approach is distinct from the other ones reported in the literature to-date, as it tends to simplify the RNN's learning, working directly at the signal level and avoiding relevant changes in network architecture and learning techniques. The overall system (called the recurrent multiscale network, RMN) is described and its performance tested through typical tasks, namely the latching problem and time series prediction.

Published in:

Circuits and Systems, 2003. ISCAS '03. Proceedings of the 2003 International Symposium on  (Volume:5 )

Date of Conference:

25-28 May 2003