Scientific-Workflow-Management-as-a-Service in the Cloud

Scientific workflow management systems have been around for many years and provide essential support such as management of data and task dependencies, job scheduling and execution, provenance tracking, etc. to scientific computing. While we are entering into a “big data” era, it is necessary for scientific workflow systems to integrate with Cloud platforms so as to deal with the ever increasing data scale and analysis complexity. In this paper, we present our experience in offering the Swift scientific workflow management system as a service in the Cloud. Our solution integrates Swift with the OpenNebula Cloud platform, and supports workflow specification and submission, on-demand virtual cluster provisioning, high-throughput task scheduling and execution, and efficient and scalable Cloud resource management. We demonstrate the capability of the solution using a NASA MODIS image processing workflow.