Exploring Enhanced Dominant Resource Fairness Using Linear Programming Calculated Weights

Maintaining resource fairness while achieving optimization for various performance metrics such as resource utilization, turnaround time and job latency is a well-known resource scheduling challenge in cloud computing. Despite the significant progress made with the introduction of dominant resource fairness by Ghodsi et al., which ensures major allocation properties such as sharing incentive, strategy-proofness, envy-freeness and Pareto efficiency to be achieved, in practice, strict adherence to fairness may still lead to non-optimized utilization of resources when a series of jobs are scheduled on a shared computing infrastructure. On the other hand, prioritizing optimization without consideration for fairness is not acceptable. Therefore, there is a need for an approach that can address both fairness and optimization. In this paper, we present LPWD, an enhanced Dominant Resource Fairness based scheduling approach that utilizes linear programming based calculated weights to balance fairness with user-specific configurable optimization objective. LPWD maintains relative and proportional fairness and employs linear programming to conduct dynamic resource allocation planning for arriving jobs on-the-fly and generate weights for each job based on a multi-objective optimization. These calculated weights are then applied to the weighted dominant resource fairness policy that guides the reordering of the jobs in the queue before deployment to the cloud. We have implemented and evaluated LPWD in a Kubernetes cluster hosted on the Chameleon cloud infrastructure, which is a configurable platform widely used by scientific communities. Our workloads are based on normalized Google trace data and Alibaba trace data and capture characteristics of resource demands in a real-world production setting. The experimentation with LPWD shows that it leads to resource utilization improvement up to 7.62% and overall turnaround time improvement up to 8.21% while maintaining job latency.