Scheduling Big Data Workflows in the Cloud under Deadline Constraints

With the advent of cloud computing, an unbound number of compute resources can be leased from the cloud providers. In such an environment, the number of assigned resources to a workflow can be elastically scaled in and out on a demand basis using the added Quality of Service (QoS) constraints such as the budget and the deadline. The heterogeneous nature of the cloud resources makes the decision of selecting resource type for each workflow a challenging problem. Although there are several existing research studies that propose both static and dynamic scheduling algorithms for both homogeneous and heterogeneous cloud resource types, they do not take advantage of the data dependency information that is part of the workflow structure during the scheduling process. There is still room for improvement, since the scheduling problem is an NP-hard problem. In this paper we propose a new Big data wOrkflow scheduleR undeR deadlIne conStraint (BORRIS) that is used to minimize the execution cost of the workflow under a provided deadline constraint in a heterogeneous cloud computing environment. We have implemented the proposed algorithm in our big data workflow system called DATAVIEW and the experimental results show the competitive advantage of our approach.