Understanding Software Platforms for In-Memory Scientific Data Analysis: A Case Study of the Spark System

Over the last five years, Apache Spark has become a major software platform for in-memory data analysis. Acknowledging its widespread use, we present a comprehensive study of system characteristics of Spark with a focus on scientific data analytics performing large-scale matrix operations. We compare its performance to SciDB, a disk-based platform for array data analysis. A benchmark, ArrayBench, is developed to evaluate the performance of matrix processing for scientific data analytics. ArrayBench is applied to data from a real biological workflow whose data inputs are in matrix form. Herein, we report the findings, which shed light on the improvement of Spark and SciDB and future development of large-scale scientific data analytics.