Language-Guided Progressive Query Refinement With Diffusion Model for Visual Grounding on Remote Sensing Images

Remote sensing visual grounding (RSVG) is a lively research topic in the RS community. Due to the superior performance of transformers in cross-modal interaction and alignment, they have dominated the RSVG community, and numerous transformer-based methods have been developed. Although they have achieved success in various applications, their performance in visual and textual feature alignment still does not reach the level we expected. In addition, these methods place relatively limited emphasis on bounding box coordinate generation, where directly predicting coordinates through a decoder and grounding head can result in suboptimal localization accuracy. To overcome the above limitations, this article presents a new RSVG method, named language-guided progressive query refinement with diffusion model for visual grounding (LPQR-DVG). It contains a multiscale text-guided feature learning module (MTFLM) and an iterative query localization module (IQLM). MTFLM integrates text guidance into all stages of the visual feature extraction process through cross-attention layers, effectively incorporating textual information into the visual representation. In this way, text-aware visual features can be effectively obtained, which facilitates cross-modal interaction and target localization. IQLM improves cross-modal feature alignment and iterative target localization by introducing an RSVG-oriented reversed U-Net with progressive query refinement (PQR) units. Through the PQR units in IQLM, visual and text features are progressively aligned via multiple rounds of interaction, enabling the query to capture the target information referred to by the text gradually. Finally, a diffusion-based iterative training strategy is introduced to help LPQR-DVG map the random bounding box coordinates into the final predictions progressively. The encouraging results counted on the DIOR-RSVG and OPT-RSVG datasets confirm the effectiveness of LPQR-DVG. Our source codes are available at https://github.com/TangXu-Group/LPQR-DVG