The effective dimension of the space of hidden units

The authors show how the effective number of parameters changes during backpropagation training by analyzing the eigenvalue spectra of the covariance matrix of hidden unit activations and of the matrix of weights between inputs and hidden units. They use the standard example of time series prediction of the sunspot series. The effective ranks of these matrices are equal to each other when a solution is reached. This effective dimension is also equal to the number of hidden units of the minimal network obtained with weight-elimination.<
>