Minimize Resource Cost for Containerized Microservices Under SLO via ML-Enhanced Layered Queueing Network Optimization

The microservice based architecture is widely used in large software systems. Despite its profound advantages, introduction of microservices brings in many challenges, especially in terms of autoscaling. Layered Queueing Network (LQN) as an effective mathematical formation for representing distributed systems can be used for modeling microservices and predicting microservices' performance. Traditional analysis methods solving LQN requires product-form solution and FCFS service discipline. Complex software design and unstable container environment is likely to prevent microservices from meeting these assumptions. To address this problem, we propose a novel autoscaling method using machine learning enhanced LQN optimization to minimize the total resource cost while satisfying the the requirement of Service Level Objectives (SLO). We adopt Deep Neural Networks (DNNs) to fit the submodels from LQN decomposition and propose a DNN based Method of Layers (MOL) to obtain the average response time metric. It avoids the drawbacks of the existing analytical methods. Applying particle swarm optimization to the DNN enhanced LQN, we present an effective method to search for the optimal resource allocation to containerized microservices that achieves our goal.. Experimental results show that our autoscaling method can reduce total CPU costs by 30% to 50% against baselines.