Task Scheduling in Multi-Cloud Environments for Spark Workflow under Performance Uncertainty

To fulfill their expanding computational demands, businesses are using cloud computing more and more these days. However, cloud systems alone may not always suffice. Consequently, multi-cloud systems, which provide more scalable storage and computing resources, are becoming more popular. This paper focuses on scheduling Spark workflow tasks in a multi-cloud environment. It addresses the challenges posed by different pricing models, dynamic resource provisioning, inter and intra transmission time, and the instability of resource performance. To tackle these issues, in this work, we propose a heuristic-based solution that considers factors such as VM instances, precedence constraints, transmission times, and the impact of performance uncertainty aiming to minimize rental costs while ensuring that workflow deadlines are met. The results show that the proposed method is effective in scheduling Spark workflow tasks in a multi-cloud environment while considering performance uncertainty.