RL-ISegNet: Refining Instance Segmentation for Remote Sensing Imagery via Iterative Reward Maximization With Limited Samples

Instance segmentation of remote sensing imagery (RSI) is vital for applications like geographic information system (GIS) updates and urban planning. Due to RSI’s diversity (e.g., scale variations and complex object shapes), instance segmentation models require extensive labeled data, which is costly and labor-intensive. To mitigate this dependence on extensive labeled data, we propose a reinforcement learning (RL)-driven iteratively-refined instance segmentation model for RSI by RL (RL-ISegNet) framework, which uses policy gradient optimization to enhance the learning of hard samples in limited labeled data, thereby improving data utilization efficiency under limited samples. Specifically, our RL-ISegNet defines a sequential optimization process for hard samples, using cumulative rewards from each subtask (region proposal, detection, and segmentation) to provide future feedback guidance for optimizing network parameters. Meanwhile, considering the conflicts between subtasks, we propose a metric to quantify the positive or negative impacts between subtasks as a loss constraint, thereby increasing the sampling probability of trajectories with positive impacts. Additionally, we developed a Fisher information-based method to freeze low uncertainty parameters, reducing the training time. The experimental results show that our RL-ISegNet improves data utilization efficiency by +12.2% and achieves improvements of +7.1% in instance segmentation metrics with 4% of the training dataset.