Online VM Service Selection with Spot Cores for Dynamic Workloads

Over the past ten years, many different approaches have been proposed for different aspects of the problem of cost-effective resources management for long running, dynamic and diverse workloads such as processing query streams or distributed deep learning. Particularly for applications consisting of containerized microservices, researchers have attempted to address problems of dynamic selection of, for example: types and quantities of virtualized services (e.g., VMs, serverless functions, data-storage), vertical and horizontal scaling of different mi-croservices, assigning microservices to VMs, task scheduling, or some combination thereof. Herein focusing on selection decisions of on-demand VM services, we consider the problem of creating and actively maintaining a training dataset for supervised machine-learned frameworks like deep neural networks and more light-weight, adaptable online optimization frameworks. For both decision frameworks, we make a case for the usefulness of spot cores and incremental search techniques like simulated annealing to reduce workload preemption while searching the decision space to explore the trade-offs between service-level objectives (SLOs) and cloud-spend. Based on user input, a macroscopic objective that captures both performance and cost will be used. We are particularly interested in scenarios with complex workloads and cloud-service offerings that vary over time.