Large-Scale Service Mesh Orchestration With Probabilistic Routing in Cloud Data Centers

Service mesh architectures are emerging as a promising microservice paradigm for developing online cloud applications. However, in large-scale microservice scenarios, frequent service communications, intricate call dependencies, and stringent latency requirements bring great pressure to efficient service mesh orchestration. In this case, the problems of service deployment and request routing based on service mesh architectures are tightly-coupled and interdependent, and cannot be effectively optimized individually, enlarging the difficulty for collaborative orchestration. When microservice multiplexing, parallel dependencies, and multi-instance modeling are considered, the difficulty is further aggravated. Nonetheless, most existing work failed to propose appropriate models and methods for the above challenges. Therefore, this article studies the large-scale service mesh orchestration with probabilistic routing and constrained bandwidths for parallel call graphs. We leverage the open Jackson queuing network theory to capture crucial microservices and analyze request processing, queuing, and communication latency for massive user requests in a fine-grained way. Then, this article proposes an efficient three-stage heuristic, which achieves elegant multi-instance consolidation and probabilistic multi-queue routing to reduce response latency and cost. We also provide the algorithm complexity and mathematical analysis of the performance. Finally, extensive trace-driven experiments are performed to validate the superiority of our proposed algorithm over other baselines.