Architecting Cloud Workflow: Theory and Practice

The data scale, science analysis and processing complexity in scientific community are growing exponentially in the "big data" era. Cloud computing paradigm has been widely adopted to provide unprecedented scalability and resources on demand, while scientific workflow management systems (SWFMSs) have been proven essential to scientific computing and services computing. Uniting the advantages of both cloud computing and SWFMSs can bring a valuable solution to the scientific "big data" problem to researchers. Although a series of work have concentrated on integrating SWFMSs with Cloud platforms that provide much experience for future research and development, a study from an architectural perspective is still missing. The main contributions of this paper are: 1) based on a comprehensive survey of the available integration options, we propose a service framework for integrating SWFMSs with Cloud computing, 2) we implement the service framework based on various Cloud platforms to validate the feasibility of the proposed framework, and 3) we conduct a set of experiments to demonstrate the capability and use a NASA MODIS image processing workflow as a showcase of the implementation.