Dynamic K Mechanism for Accelerating Speculative Decoding in vLLM

With the rapid growth in parameter scale of large language models (LLM), the computational cost during their inference phase has become a major bottleneck for deployment. Although speculative decoding serves as a lightweight acceleration technique—generating candidate tokens via a draft model and verifying them through the main model—it effectively reduces latency. However, mainstream implementations typically adopt a fixed generation step-size K, failing to adapt to diverse input scenarios and thus limiting the upper bound of decoding performance. To address this issue, this paper proposes a dynamic K-based speculative decoding method that adaptively adjusts the generation step-size according to draft quality, and implements it for the first time in the vLLM inference framework. Experimental results demonstrate that this strategy significantly enhances the throughput ceiling of speculative decoding without compromising output quality, providing a viable system optimization path for efficient general-purpose inference engines.