A Switch Criterion for Hybrid Datasets Merging on Top of Map Reduce

With MapReducepsilas restricted structure, multi-datasets merging problem, commonly in many data mining applications, cannot be efficiently resolved with MapReduce. This paper proposes a novel hybrid datasets merging algorithm on top of Map Reduce, HDMA. HDMA can help to automatically determine the relatively better one between two methods, DMCM and DPM, which have different effective fields. HDMA retains the advantages of both methods, and it can make good use of the memory of data nodes. Experiments show that HDMA can get best performance in most situations.