Efficient GPU Resource Pooling for Machine Learning Workloads

General Purpose Computing on GPU (GPGPU) has become popular in the recent decade as it enabled significant faster computation speeds compared to traditional CPU based computing. Tasks like training machine learning models can be accelerated significantly by using GPUs. Current commercial cloud computing resources with GPUs incur considerably high cost for users and there is no efficient mechanism for personal GPU holders to rent their GPUs for GPGPU tasks. This paper presents a GPU resource pooling system, designed to address the inefficiencies in current GPU allocation mechanism which leads to resource underutilization and increased costs, by enabling dynamic sharing of idle GPUs for executing one-to-one and distributed machine learning tasks. The system operates on a pay-as-you-go cost model, optimizing GPU usage and ensuring cost efficiency. Through a dynamic scheduling algorithm, the system intelligently allocates GPU resources, enhancing workload distribution and reducing execution times. Built with scalability in mind, the system’s architecture can be extended to support various general-purpose computing tasks in the future. Our evaluations demonstrate significant improvements in resource utilization, making this solution a practical and cost-effective choice for decentralized GPU computing.