Entropy4Cloud: Using Entropy-Based Complexity to Optimize Cloud Service Resource Management

In cloud service resource management system, complexity limits the system's ability to better satisfy the application's quality of service requirements, e.g., cost budget, average response time, and reliability. Numerousness, diversity, variety, uncertainty, etc., are some of the complexity factors that lead to the variation between expected plan and actual running performance of cloud applications. In this paper, after defining the complexity clearly, we identify the origin of complexity in cloud service resource management system through the study of “Local Activity Principle.” In order to manage complexity, an entropy-based methodology is presented to use, which covers identifying, measuring, analyzing, and controlling (avoid and reduce) of complexity. Finally, we implement such idea in a popular cloud engine, Apache Spark, for running analysis as a service. Experiments demonstrate that the new entropy-based resource management approach can significantly improve the performance of spark applications. Compare with the fair scheduler in Apache Spark, our proposed entropy scheduler is able to reduce overall cost by 23%, improve the average service response time by 15-20%, and minimized the standard deviation of service response time by 30-45%.