Workload Scheduling on Heterogeneous Devices

Hardware accelerators have become the backbone of many cloud and HPC workloads, but workloads tend to statically choose accelerators leaving devices unused while others are oversubscribed. We propose a holistic framework that allows a computational kernel to span across multiple devices on a node, as well as multiple applications being scheduled on the same node. Our work sharing and co-scheduling framework allows kernels to be migrated between devices, expand to span more devices, or contract to fewer devices. The scheduler can make these decisions dynamically based on a pluggable scheduling algorithm in order to optimize for different objectives, e.g., job throughput, job priorities or some hybrid. Experiments on a CPU+GPU+FPGA platform indicate speedups of 2.26X over different applications and up to 1.25X for co-scheduled workloads over baselines. Besides performance, a major contribution of our work lies in ease of programmability with a single code base compiled and runtime controlled across three vastly different execution devices.