BIVD: Improved Side Adapter Network based on BiFormer

Currently, most open vocabulary models rely on CLIP (Contrastive Language-Image Pre-training) to classify masked regions, but CLIP models still have many drawbacks. The traditional transformer model is context aware through self attention, but this local context based mechanism is limited by the sequence length of information, it can also impact model effect. Inspired by the article BiFormer, This article replaces the traditional transformer model in the Side Adapter Network (SAN) model with a variant module BIVD block based on BiFormer blocks. This model follows the latest transformer model and extracts features using a four layer feature pyramid structure, For the reduced spatial resolution caused by patch merging, add a Visual-Decoder to reshape the visual vector sequence into a spatial feature map, and restore it to the original resolution through convolution and upsampling. This can enhance the extracted feature information through a feature pyramid without affecting the fusion with CLIP layer features. But the cost is increasing the cost of the model. This article was tested on 5 commonly used datasets and showed a stable improvement in accuracy.