MVFG: Enhancing Semantic Segmentation via Mix Vision Transformer With a Feature Guidance Module Powered by SAM Masks

The challenges in semantic segmentation of remote sensing images arise primarily from significant variations in target sizes, diverse shapes, and complex background environments. To address these challenges, we have developed a feature-guided semantic segmentation model for remote sensing images, further enhancing its accuracy with high-fidelity masks generated by the segment anything model (SAM) vision large model. The principal contributions of this study are outlined as follows: We introduce a semantic segmentation model that employs feature guidance, ingeniously integrating convolutional neural network (CNN) and Transformer architectures to effectively capture both the local details and the global context of remote sensing images. We propose an improved method based on the SAM vision large model, featuring a dual-branch structure that incorporates two key modules: semantic voting (SV) and mask refinement. The SV module corrects semantic errors in both branches. Subsequently, an MR module is utilized to further enhance these masks. In experimental validations using the iSAID dataset, the model significantly improved the mean Intersection over Union (mIoU) to 61.54% and the mean $F1$ -score to 74.69%, underscoring the significance and applicability of the proposed methodology.