An Apache Spark Implementation for Text Document Clustering

As the volume of data generated and stored on a daily basis is constantly increasing, the need for finding techniques in terms of the automated discovery of information from them has arisen. This purpose can be effectively solved with the use of text mining, which uses methods derived from data mining, information retrieval, machine learning, as well as natural language processing. This paper addresses the problem of extracting textual information from large collections of documents by efficiently exploiting clustering techniques in a cloud computing infrastructure. The clustering was performed using three different algorithms, namely k-Means, Bisecting k-Means, and Gaussian Mixture Model (GMM). To evaluate the quality of these methods, we experimented in the Apache Spark distributed environment, on several well-known datasets, the documents of which have been manually clustered.