Vision Transformers with Cross-Attention Pyramids for Class-Agnostic Counting

Counting unseen objects in images remains a challenging issue in computer vision. Additionally, existing methods often stumble on localization and scaling quirks, hindering their performance. We introduce a novel vision transformer with cross-attention pyramids (CAP) that tackles this challenge head-on. CAP features two key innovations: a dual-branch architecture with interconnected cross-attention, amplifying feature pyramids for robust object representations, and a dedicated feature aggregator module that further improves performance. The proposed method prevents confusion with challenging backgrounds, avoids over-counts, and leads to accurate counts for objects of various scales. On the FSC-147 test set, CAP surpasses the current state-of-the-art method by 16% in MAE on the few-shot setting and also shows improvement on the zero-shot setting.