Understanding the Workload Characteristics in Alibaba: A View from Directed Acyclic Graph Analysis

Cloud computing technology is widely used in data centers due to the benefits such as high scalability, high availability, on-demand provision, and so on. Understanding the characteristics of complex workloads is crucial for job scheduling, capacity planning, and performance optimization, which provide guarantees for the normal operation of cloud-native applications. Co-location of online services and batch jobs is a common way to improve resource utilization in today's cloud data centers. It is very important for the resource scheduler to adapt to the characteristics of complex and varied workloads. In this paper, we perform an in-depth analysis on the latest released trace dataset by Alibaba in December 2018, consists of 4195049 batch jobs and 71476 containers co-locating on about 4000 machines. Different from Alibaba trace 2017, the information of directed acyclic graph (DAG) is added in the trace, which explains the relationship between the jobs and tasks. We first show the basic statics of the cluster. Then, we conducted an in-depth analysis of the workload types and DAG information. The results reveal several new insights: i) The resource utilization in the cluster shows an obvious periodic phenomenon (day and week). ii) There is a resource (e.g. CPU, memory usage) overprovisioning phenomenon in the cluster. iii) The workloads' characteristics of different workload types are very different. iv) The distribution of instance startup time and termination time is unbalanced. v) 90% of the job's DAG length does not exceed 30 while the maximum length is 54.