Aligning Pattern Extraction Algorithms for the Lambda Architecture

For quite some time now, data have become the new oil of the digital industry. The spread and evolution of information technologies and connectivity between people and devices have enabled a new dimension of big-data storage and analytics that could bring major improvements across industries. In this paper, we propose a new, frequent itemset mining approach. The challenge is to apply traditional extraction techniques in a distributed environment. The main originality of our mining method is to take benefits of a performant existing algorithm and use a novel data structure to maintain frequent sequential patterns coupled with a quick pruning strategy. The proposed approach has been implemented using Spark to further improve the efficiency of iterative computation. Numeric experiment results using standard benchmark datasets by comparing the proposed algorithm with the existing algorithm, FP-Growth, demonstrate that our approach has better efficiency and scalability.