An Improved YOLOv8 method for Ship Instances Segmentation of Small targets in Remote Sensing Images

Since traditional instance segmentation methods have low accuracy and slow speed for small target in remote sensing images, an improved YOLOv8 method is proposed to improve the detection accuracy of small targets. Firstly, we adopt a dual-level routing attention module based on the Vision Transformer architecture to effectively capture long-range depen-dencies among targets and detailed information within individual targets. This facilitates a better understanding of interconnections among small-scale target ships, consequently leading to enhanced accuracy and robustness in the segmentation process. Secondly, Wise IoU Loss function based on dynamic nonmonotonic focusing mechanism is used. By introducing a weight map to dynamically adjust the model's precise matching of target boundaries, the robustness of the model is improved and missed detections caused by small target sizes are avoided. Finally, a small object detection layer is added to connect shallow and deep feature maps for multi-level information fusion, making the network more focused on small object detection. The results show that the improved YOLOv8 method improves the AP by 2.5% compared with the baseline model (YOLOv8) on the HRSID dataset. In contrast to other state-of-the-art instance segmentation methods such as Mask RCNN and Mask Scorning RCNN, the AP of the improved method increased by 5.8% and 5.4%, respectively. Experimental results show that the modified method significantly improves the segmentation performance of small objects in remote sensing images.