Evaluating a Data-Aware Scheduling Approach to Reduce Processing Costs of DMCF Workflows

As scientific data analysis applications become more and more complex, there is a great need to simplify the definition and execution of such applications, particularly when dealing with large datasets. The Data Mining Cloud Framework (DMCF) is a system allowing domain experts to design and execute complex data analysis workflows on cloud platforms, relying on cloud storage services for every I/O operation. In order to enhance I/O operations, we propose the integration of DMCF with Hercules, an in-memory I/O solution that can be used in combination with DMCF as an alternative to cloud storage services to improve the I/O performance of workflow executions. The integration between DMCF and Hercules is based on a data-aware scheduler that exploits data locality and in-memory I/O to reduce run time of workflows. The goal of this work is to evaluate the reduction of processing costs obtained by using the proposed data-aware scheduler, compared to the costs obtained with the original DMCF solution on the same workflow. The evaluation performed on a 32-node cloud cluster results in 52% reduction of I/O time, which results in 8% total execution time reduction, and 9% of cloud services cost reduction.