An Online Algorithm for Inference Service Scheduling Using Combinations of Server-Based and Serverless Instances in Cloud Environments

With the continuous development in the field of machine learning, there is an increasing demand for the cloud-based machine learning inference services, which are latency-sensitive tasks, such as the service requests from the Internet of Things (IoT) devices. These inference services are generally accompanied by fluctuations and uncertainties, so they often require vastly varied numbers of servers at different time spots. As a result, how to dynamically and rationally schedule cloud servers for inference services has become an important issue. Alibaba Cloud currently provides its serverless instances called elastic container instance (ECI), and due to the advantages of pay-as-you-go billing and second-level elasticity, they are well-suited for handling bursty or fluctuating workloads. At the same time, Alibaba Cloud’s subscription-based elastic compute service (ECS) instances can be used for steady-state workloads. Our objective is to dynamically combine these two types of instances to deal with inference service requests. In this article, we propose a deterministic online algorithm that can rationally schedule these two types of instances to optimize costs without requiring knowledge of future workloads. We prove that the proposed online algorithm achieves a competitive ratio of no more than 2 compared to the optimal offline algorithm. Through simulation experiments, we demonstrate that our algorithm outperforms three benchmarks, which are all-reserved algorithm, all-on-demand algorithm, and traditional online algorithms that only use ECS instances. Our algorithm exhibits superiority across various workloads and can significantly reduces costs in most cases.