History Tracker: Retrieving Historical Image Embeddings for Efficient Fine-Grained Reasoning in Vision-Language Models

Vision-Language Models (VLMs) effectively align images and text, but they often struggle with fine-grained reasoning tasks. Fine-grained training also typically demand substantial GPU memory and a large number of trainable parameters. While existing Parameter-Efficient Fine-Tuning (PEFT) methods address these challenges, they also introduce new limitations, such as compromising generalization performance with additional parameters and causing overfitting with partial fine-tuning. To overcome these limitations, we propose History Tracker, a novel PEFT method designed to enhance fine-grained reasoning performance in VLMs. History Tracker leverages the stacked architecture of the image encoder, dividing its layers into shallow, middle, deep, and global groups. It then retrieves and compresses embeddings from these groups into compact tokens, reinforcing the distribution of image embeddings and allowing the text decoder to capture fine-grained details. Furthermore, we introduce a fine-grained dataset to fine-tune and evaluate models. This dataset focuses on tasks where highly similar text needs to be ranked based on its alignment with the given image. Experimental results demonstrate that History Tracker outperforms existing fine-tuning methods on both general benchmarks and fine-grained reasoning tasks. By fine-tuning only 7.03% of parameters, our method strikes a balance between fine-grained reasoning, generalization, and efficiency.