CLIP-MoA: Visual-Language Models With Mixture of Adapters for Multitask Remote Sensing Image Classification

In recent years, the rapid growth of remote sensing (RS) datasets has greatly enhanced the difficulty of RS image classification tasks. Given the remarkable success of large vision-language models (VLMs) in various domains, their potential for RS image classification tasks has attracted significant attention. In this study, we focus on contrastive language-image pre-training (CLIP), a widely recognized contrastive learning model known for its strong zero-shot transferability. However, the domain gap between the pre-training data (e.g., natural images) of CLIP and RS data can significantly hinder the application of CLIP. First, the scarcity of labeled RS data poses a significant challenge for fine-tuning CLIP. Second, the complexity and diversity of RS images substantially increase the fine-tuning time and computational resources needed to individually adapt CLIP to new tasks. Third, independently fine-tuning CLIP for each RS dataset ignores the inter-task relationships that frequently exist among RS images from various sources, leading to reduced classification accuracy. To address these challenges, we propose a learnable mixture of adapters (MoAs) module in the visual branch of CLIP, resulting in the CLIP-MoA framework. Inspired by the mixture of experts (MoEs), the MoA module models inter-dataset relationships by integrating multiple adapters through a gating network, while preserving original information via a residual connection between the MoA input and the fine-tuned features. This module enables CLIP-MoA to effectively capture both inter-task commonalities and task-specific characteristics by fine-tuning a minimal set of MoA parameters across multiple datasets simultaneously, thereby reducing computational overhead and fine-tuning time while improving classification accuracy. Comprehensive experiments conducted on a multitask benchmark involving 11 RS image classification datasets demonstrate that CLIP-MoA achieves significant improvements in both classification accuracy and generalization capability compared to zero-shot CLIP, linear probe CLIP, CLIP-Adapter, and CoOp.