Crowd: An KV Cache Eviction Policy Which Uses Crowd Information to Select Evicted Key-Value Pairs

Although large language models have been widely applied in various aspects of daily life, it is worth noting that, in addition to their substantial demand for computational resources, they also require an exceptionally large amount of GPU memory. This is because, aside from the model weights, the key-value cache must also be stored in GPU memory, and the size of this cache increases with the batch size and sequence length, becoming a significant bottleneck in inference. In this paper, we introduce an innovative method for optimizing key-value (KV) cache, significantly reducing its memory usage. Through experimental validation, we observed that the higher the similarity between query vectors, the higher the similarity between the key vectors they attend to. Additionally, experiments showed that the vectors most similar to the current query vector are concentrated within its local window size. Based on these findings, we propose a new KV cache eviction policy called Crowd, which can dynamically select the necessary key-value pairs for inference without requiring model retraining or fine-tuning. Extensive experiments demonstrate the superiority of Crowd during both the pre-filling and autoregressive decoding stages.