Automatic extractive text summarization using K-means clustering

The rise in the dimension of the World Wide Web has made an explosion of the amount of accessible information. As the textual data involves several instances of redundancy, omission of part of sentences or entire sentences is possible without altering the meaning of the document. Summarization of the text can informally be defined as the act of condensing the document from its original size without significantly compromising the semantics. For the purpose of generating an appropriate summary, the raw text is first pre-processed which involves - removing nonASCII characters and stop-words, tokenizing and stemming. Appropriate features are extracted from the data, tf-idf values for each word are computed and the entire pre-processed data is then transformed into a tf-idf matrix. Every sentence of the document will be represented as a vector in the dimensional space of the document's vocabulary. To obtain a concise summary, sentences are appropriately clustered based on the degree of separation of vectors in the Euclidean place. Association of sentences to a cluster using K-means method is totally based on cosine similarity. The count of the clusters is to be formed is predefined. As the number of clusters increase the accuracy of the summary increases. From each of the clusters the sentences which are informative are picked to form the final summary. Using recall and precision measures, the effectiveness of the summary is verified.