An improved partition algorithm to find large itemsets in K-dimensional space

It has been shown that the current partition algorithm suffers from generating a lot of candidate itemsets when the search range is large. Now an improvement algorithm is proposed to find the large itemsets in the database without the constraint of the maximal range. In the improved algorithm, “divide and conquer” approach is used to recursively partition the whole Hadoop file until a disk bucket is reached. At that time, all the large itemsets can be found in a bucket and then two equal-sized partitions are merged. Finally, all the large itemsets in the Hadoop file can be found. The Hadoop file can be used to store the transactions containing the large items such that only the regions in processing are needed to be scanned, which reduces the unnecessary database scan. The experiment shows that the algorithm is better than the traditional algorithm. Also, sensitivity analysis on four parameters is analyzed.