ExpertDRL: Request Dispatching and Instance Configuration for Serverless Edge Inference With Foundation Models

The prevalence of the pre-training & fine-tuning paradigm enables machine learning models to quickly adapt to various downstream tasks by fine-tuning pre-trained foundation models (FMs), greatly facilitating various IoT applications that rely on model inference in dynamic edge serverless environments. Efficiently dispatching inference requests and configuring instances to batch inference requests can significantly enhance resource efficiency. However, existing serverless inference solutions are tailored for traditional models, make coarse-grained request dispatching and instance configuration decisions, fail to exploit the shared model backbone characteristics of the FM and capture delayed rewards in dynamic environments, and ignore communication latency between edge sites, resulting in high costs and constraint violations. In this paper, we leverage our insight that fine-grained batch inference requests can effectively exploit the shared model backbone feature of FM to save monetary costs. We propose an algorithm that incorporates deep reinforcement learning (DRL) and expert intervention for fine-grained request dispatching and instance configuration, where the DRL component outputs fractional solutions as guidance, while the expert intervention module integrates our insights—batching reduces monetary costs at the expense of increased inference latency, whereas higher configurations shorten inference latency. This module rounds fractional solutions and adjusts instance configurations to search for optimal solutions while satisfying constraints, with theoretical guarantees rigorously proved. Finally, we conducted our experiments on an OpenFaas-based platform and simulator, and extensive trace-driven evaluation results show that ExpertDRL can save costs by up to 85.14% and improve request acceptance ratio by up to 26.93%, compared to the state-of-the-art solution.