RingMoGPT: A Unified Remote Sensing Foundation Model for Vision, Language, and grounded tasks

Recently, multi-modal large language models (MLLMs) have shown excellent reasoning capabilities in various fields. Most of the existing remote sensing MLLMs solve image-level text generation problems (e.g., image captioning), but ignore the core issues of object-level recognition, location and multi-temporal changes in the field of remote sensing. In this paper, we propose RingMoGPT, a multi-modal foundation model that unifies vision, language, and localization. Based on the idea of domain adaption, RingMoGPT can complete training by fine-tuning only a few parameters. To make the model capable of object detection and change captioning, we further propose a Location- and Instruction-aware Q-Former and a change detection module, respectively. To improve the performance of RingMoGPT, we carefully design the pre-training dataset and the instruction-tuning dataset. The pre-training dataset contains over a half million high-quality image and text pairs, which are generated through a low-cost and efficient data generation paradigm. The instruction-tuning dataset contains more than 1.6 million question-answer pairs, including 6 downstream tasks: scene classification, object detection, visual question answering, image captioning, grounded image captioning, and change captioning. Our experiments show that RingMoGPT performs well on 6 tasks, especially its ability to analyze multi-temporal data changes and identify dense objects. We also verified the model under zero-shot setting, and the results show that the proposed RingMoGPT also has good generalization ability in the face of new data.