Scaling and Parallelization of Big Data Analysis on HPC and Cloud Systems

Big data analysis can exhibit significant scaling problems when migrated to High Performance Computing (HPC) clusters and/or cloud computing platforms if traditional software parallelization techniques such as POSIX multi-threading and Message Passing Interface (MPI) are used. This paper introduces a novel scaling technique based on a-well-known array job mechanism to enable a team of FDA researchers to validate a method for identifying evidence of possible adverse events in very large sets of patient medical records. The analysis employed the widely-used basic Statistical Analysis Software (SAS) package, and the proposed parallelization approach dramatically increased the scaling and thus the speed of job completion for this application and is applicable to any similar software written in any other programming language. The new scaling technique offers O(T) theoretical speedup in comparison to multi-threading and MPI techniques. Here T is the number of array job tasks. The basis of the new approach is the segmentation of both (a) the big data set being analyzed and (b) the large number of SAS analysis types applied to each data segment. The large number of unique pairs of data set segment and analysis type segment are then each processed by a separate computing node (core) in pseudo-parallel manner. As a result, a SAS big data analysis which required more than 10 days to complete and consumed more than a terabyte of RAM on a single multi-core computing node completed in less than an hour parallelized over a large number of nodes, none of which needed more than 50 GB of RAM. The massive increase in the number of tasks when running an analysis job with this degree of segmentation reduces the size, scope and execution time of each task. Besides the significant speed improvement, additional benefits include fine-grained checkpointing and increased flexibility of job submission.