Data-intensive Workflow Execution using Distributed Compute Resources

Cloud computing has become a necessary utility for scientific and technical applications. Many diverse web services are published and subscribed using cloud data centers. It has become fairly easy to use services from Cloud Service Providers (CSPs) for computation and data processing. However, even with all their benefits, commercial cloud resources are not economical when large data processing is required. Hence, educators and researchers need guidance to use commercial cloud resources to run large data processing workflow applications within a budget. In this paper, we propose a framework to help users to leverage distributed compute resources to execute data-intensive application workflows, under budget constraints. We demonstrate how our framework can be used by users who may have access to small-scale compute resources in-house, to seamlessly interoperate with public cloud resources.