DAG Constrained Scheduling Prototype for an Astronomy Exa-Scale HPC Application

The Square Kilometer Array (SKA) under construction aims to be the world's largest telescope. Its Science Data Processing (SDP) is responsible for processing the observed data into science-ready data products. The exa-scale data throughtput calls for scheduling to assign computation tasks to networked computing island to relieve computation pressure. Popular scheduling methods in the classic high performance computing centers cannot fit into perplexing constraints over the computation tasks due to evolving telescope management policies and a large number of users across the world. Aiming at using the SDP workflow to construct radio signal images, we propose a set of comprehensive strategies, Genetic Algorithm (GA) optimization and Critical-path Aware Earliest Finish Time (EFT), to reduce the overall execution time on different laxity time scenarios. If the observation tasks are planned in well advance, GA, which optimizes the scheduling results through a heuristic manner, is verified to outperform Round Robin. But if observation tasks are issued in short notice, to reduce time consumption, Critical-path Aware EFT is implemented. It is partitioned into upper and lower level to be a hierarchical scheduler and this mechanism provides both coarse and fine grained scheduling which guarantees the communication within the scheduler. Implemented on the popular cloud computing simulator CloudSim with datasets of different scales, Critical-path Aware EFT proves to be an effective and extendable method.