Simple recurrent networks and random indexing

We first show that the dendrogram depicting a lexical hierarchy among words that Elman obtained by training an SRN (simple recurrent networks) is in fact obtained without training the SRN. We then show that the reason why training was not required is that the SRN itself (1) assigns a random code, which is in fact a set of weights of the SRN, to each word and (2) assigns a composite code reflecting contexts (a set of words) of a word, which is in fact a vector of hidden unit activations of the SRN, to the word, since a lexical hierarchy among words can be built upon the similarity of their contexts. We ascertained that the above scheme is valid although the codes are skewed by non-linear output function (the standard sigmoidal function). We also note that the coding scheme is similar to the random indexing proposed by Kanerva and his group.