By Topic

dSimpleGraph: A Novel Distributed Clustering Algorithm for Exploring Very Large Scale Unknown Data Sets

Sign In

Cookies must be enabled to login.After enabling cookies , please use refresh or reload or ctrl+f5 on the browser for the login options.

Formats Non-Member Member
$31 $13
Learn how you can qualify for the best price for this item!
Become an IEEE Member or Subscribe to
IEEE Xplore for exclusive pricing!
close button

puzzle piece

IEEE membership options for an individual and IEEE Xplore subscriptions for an organization offer the most affordable access to essential journal articles, conference papers, standards, eBooks, and eLearning courses.

Learn more about:

IEEE membership

IEEE Xplore subscriptions

3 Author(s)
Li Lu ; Dept. of Comput. Sci., Univ. of Illinois at Chicago, Chicago, IL, USA ; Yunhong Gu ; Grossman, R.

Some of the major challenges in current clustering applications include: some data sets are so huge that it is difficult to load the entire data sets into memory for clustering, the data sets are often distributed over different locations for various reasons, which makes it impossible to process them centrally, and when lacking prior knowledge of the unknown data sets, it is troublesome to choose the appropriate parameters to feed into existing clustering algorithms. Therefore, a distributed clustering algorithm without too many parameters becomes rather appealing. Although some distributed clustering algorithms have been proposed, it is still a challenge for them to solve all of these problems. In this paper, we propose and implement a novel micro-cluster based distributed clustering algorithm called dSimpleGraph. An equivalence relation on two micro-clusters is defined. Relying on the relation, dSimpleGraph can efficiently cluster data on the local machines, moreover, it can easily generate a determined global view from local views. Only two scalar parameters are needed and the generated clusters can be any shape. Its MapReduce-style structure allows it to be easily implemented on existing distributed computing platforms. Extensive experimental studies show that dSimpleGraph is very fast and very suitable for exploring very large scale unknown data sets.

Published in:

Data Mining Workshops (ICDMW), 2010 IEEE International Conference on

Date of Conference:

13-13 Dec. 2010