Data-Driven Workload Generation Based on Google Data Center Measurements

A large dataset of workload measurements has been released by Google. The wealth of disclosed data allows a deep dive into real workload patterns. With the aim of providing tools to generate realistic workloads in a simple way, we have extracted from Google’s dataset job arrival times, number of tasks per job, required computation time, and memory of tasks. We define a statistical fitting of the relevant probability distribution, providing a simple tool to build artificial workload traces that mimic real traffic as represented by Google measurements. The workload generation algorithm is assessed by comparison of its mean response time on a test dispatching/scheduling system against the real traffic traces. In spite of being only a first-order generation model, it is shown that the proposed artificial generation can reproduce faithfully the performance of real workload in the case of large server clusters.