Beyond-Labels: Advancing Open-Vocabulary Segmentation with Vision-Language Models

Open-vocabulary semantic segmentation attempts to classify and outline objects in an image using arbitrary text labels, including those unseen during training. Self-supervised learning resolves numerous visual and linguistic processing problems when effectively trained. This study investigates simple yet efficient methods for adapting previously learned foundation models for open-vocabulary semantic segmentation tasks. Our research proposes “Beyond-Labels,” a lightweight transformer-based fusion module that uses a handful of image segmentation data to fuse frozen visual representations with language concepts. This strategy allows the model to successfully actualize enormous knowledge from pre-trained models without requiring extensive retraining, making the model data-efficient and scalable. Furthermore, we efficiently capture positional information in images using Fourier embeddings, thus improving the generalization and resulting in smooth and consistent spatial encoding. We perform thorough ablation studies to investigate the major components of our proposed method in comparison to the standard benchmark PASCAL-5i, the method performs better despite being trained on frozen vision and language characteristics.