Efficient Adaptive Batching of DNN Inference Services for Improved Latency

With the rising success of deep neural network (DNN) applications, GPU servers are increasingly utilized to provide DNN inference services. Batching, in which multiple incoming requests are grouped together for simultaneous processing, is a common practice for the DNN inference serving systems to enhance GPU utilization and thus lower the service cost. While batching enhances GPUs’ efficiency, it can extend the end-to-end latency for each individual request. This delay is especially noticeable when requests are infrequently incoming to form a complete batch. On the other hand, when the request rate is high, a consistently small batch size may not fully harness the GPU’s capacity. Handling requests one by one might decrease this latency but also raise energy consumption. Therefore, it’s crucial to strike a balance between energy efficiency and latency, optimizing the batch size to ensure timely processing without compromising performance under varying request rates. Many existing solutions aim to fully use user-set deadlines, which, even though they maximize, can heighten perceived latency, particularly in larger pipelines. In this paper, we introduce a dynamic batching strategy tailored to adapt batch sizes in real-time for varying request rates. Our method aims to balance the time taken to assemble a batch and the time required to execute it, ensuring efficient energy consumption and prompt response times. By dynamically adjusting batch size relative to the current request rate, we strive to ensure that the collection duration for a batch coincides with its execution duration. This will minimize the latency as much as possible while ensuring we utilize the batching to provide an efficient solution. We built our approach using Triton, a platform for AI model deployment and execution. Our evaluation with four different DNN models shows that our technique achieved marginally close latency as the batching deployed by the Triton inference server and improved the energy efficiency by up to 5%.