A Noval Unsupervised Learning Big Data Mining Algorithm Based on Spark Framework

The Spark framework has been widely used for the unsupervised learning of big data mining algorithms; however, it has problems. That is, the multiple data exchange iterations between master and worker nodes result in low operation efficiency. This study proposes a k-means clustering algorithm based on flora colony (FC) optimization. FC optimization was introduced to optimize the k-means clustering processes through the repeated dispersal, replication, and chemotaxis operations of the flora. Hence, the fitness of data clustering is continuously improved until the maximum number of operations or the minimum clustering accuracy threshold is reached. Then, a stable big data clustering-mining algorithm is obtained. Based on the Spark’s data partitioning framework, the proposed algorithm executes the FC-k-means algorithm on each partition to obtain local cluster centers, summarizes the results, recycles them to the master node, and runs the algorithm on the master node to aggregate all results to form the final cluster center. Simultaneously, parallel FC-k-means operations are performed on the Spark framework to obtain a better initial cluster center, which improves accuracy and convergence speeds. Experimental results show that this parallel implementation improves model accuracy and reduces runtime.