Effect of Latent Semantic Indexing for Clustering Clinical Documents

The measurement of similarity between documents is usually influenced by sparseness of term-document matrix. Latent semantic indexing (LSI) is an alternative method to solve the problem, and the dimension reduction by LSI improves the performance of the measurement of the similarity. In this study, LSI is examined as a method to cluster clinical documents containing the same clinical problems or disorders. The similarity of clinical documents was measured effectively with LSI. LSI performed better on clinical documents which can be characterized with medical terms, various expressions for the same concepts, abbreviations and typos, than editorials. Our result showed that LSI is useful for the measurement of the similarity of the clinical documents examined in this study. And the correlation between co-occurrence of terms and similarity is also analyzed as an important aspect of LSI. Not only the co-occurring terms but unshared terms between documents were found as factors influencing the similarity.