A Provenance-Aware Approach to Big Data Workflow Management in Heterogeneous Cloud Environments

Big data workflows have emerged as a powerful paradigm that enables researchers and practitioners to run complex multi-step computational processes in the cloud to gain insight into their large datasets. To create a workflow, a user logs on to a specialized software, called Big Data Workflow Management System, or simply BDW system, to select and connect together various components, or tasks, into a workflow. The workflow is then mapped onto a set of distributed compute resources, such as Virtual Machines (VMs), and storage resources, such as S3 buckets and EBS volumes. It is then executed, with different branches and tasks of the workflow running in parallel on different nodes. During execution, the BDW system captures provenance, which is the history of data derivation that describes data processing steps that yielded each output result. Workflow management, including workflow composition and schedule refinement, is a challenging problem. This problem is further exacerbated by the growing number and heterogeneity of workflow tasks and cloud resources, as well as by the growing size and complexity of workflow structures. Few efforts were made to leverage provenance for facilitating workflow composition and schedule refinement. To address these issues, we 1) produce a comprehensive conceptual model for big data workflow provenance that captures the complexity and heterogeneity of cloud-based workflow execution, 2) propose a scalable Cassandra database schema for provenance-aware workflow composition and schedule refinement, 3) outline a four-step provenance-based schedule refinement process for balancing workflow execution time and cost, and 4) present a scalable and highly available microservices-based reference architecture for big data workflow management in the cloud. Our proposed loosely coupled architecture ensures superior scalability, as well as operational and technological independence of each module within the BDW system.