Improving Efficiency of DBSCAN by Parallelizing kd-Tree Using Spark

Clustering is a data mining approach that divides data into groups that are meaningful, useful, or both. Well-known clustering algorithms include Kmeans, BIRCH, Wave-Cluster, and DBSCAN. Density Based Spatial Clustering of Applications with Noise (DBSCAN) is a density based clustering algorithm. DBSCAN is capable of identifying arbitrary shaped clusters and can eliminate noisy data. However, parallelization of DBSCAN is a challenging task because under the common technologies of MPI or OpenMP, there exists the issue of lack of fault tolerance and there is no guarantee that workload is balanced. Also, programming with MPI is a big challenge. We present a parallel DBSCAN algorithm using the new big data framework Spark. In order to reduce search time, we use the data structure kd-tree. Besides, we use a parallel approach for generation of kd-tree. Similarly, clustering of points using DBSCAN is also done in a parallel manner. We evaluate the performance as well as scalability of the algorithm and show that our implementation is better performing and scalable.