Parallel hierarchical subspace clustering for segmenting large text corpuses

Hierarchical Document Clustering helps in grouping similar documents and facilitates the subsequent extraction of useful information. High dimensionality of documents and the scale of the document-corpus often results in a heavy demand for both memory and compute-power. In this paper, we have designed and implemented a parallelization scheme for a hierarchical document clustering algorithm and also study the effect of the popular dimensionality reduction techniques on the performance of the algorithm. The parallelization scheme is amenable to be implemented on distributed parallel programming frameworks like Map Reduce. The parallelized hierarchical algorithm produces clusters of good quality as measured by the cluster-validation indices and also achieves a considerable speed-up when run on the HaLoop Map-Reduce framework.