Induced specialization of context units for temporal pattern recognition and reproduction

Additional inputs to a feedforward network, derived from the output of the hidden layer neurons, allow a feedforward network to deal with temporal pattern recognition and reproduction tasks. These 'network derived' or 'context' inputs augment the 'true' inputs to the network and allow the network to retain past information necessary for temporal sequence processing. The choice of which hidden neurons to retain to provide the context inputs is difficult. Use of all the hidden neurons increases the size of the overall network resulting in poorer generalization performance. The problem is complicated due to difficulty in choosing the number of hidden layer neurons in the first place. In this paper, we propose the use of regularization terms in the sum-of-squared error cost function. Assuming the hidden layer neurons are indexed 1,2,...,m, the regularization terms force the differentiation of hidden neurons 1 through m/sub 1/, and m/sub 2/ through m (where 1