Towards a Full Big Data Based Solution for Computationally Intensive Problems : Solving Motif Finding Problem as a Case Study

The Solution scalability of computationally intensive problems constitutes a major challenge to researcher in the High Performance Computing realm. Cloud platform services have been explored for storage and analytics purposes and taking into consideration the huge volume of data, tremendous cloud storage platforms that currently exist and provide a scalable, distributed storage service, Therefore, the challenges of implementing (MF) Motif Finding problem based on the services provided by the cloud storage platform, are addressed. In this work, Apache Hadoop is picked for Big Data processing which is an open source Platform that provides enormous number of clusters that is used for parallel implementations. MF solution was implemented using two different Big Data frameworks: MapReduce and Apache Spark, they have different implementation schemes and resources usage plans. The results are collected and analyzed against speedup value. From experiments Spark achieves much more speedup than Mapreduce. This finding is expected due to Mapreduce I/O operations overheads. The main purpose of this step is to accomplish effective parallelization and evaluates the performance of such an integrated solution.