BoxPolypSAM: Leveraging SAM in Box-Supervised Polyp Segmentation

Automatic polyp segmentation from colonoscopy images is crucial for colon cancer diagnosis. Nonetheless, training fully supervised polyp segmentation models requires timeconsuming pixel-level annotations. Instead, box annotations of polyps are more attainable. To reduce the burden of annotation, we propose BoxPolypSAM, a box-supervised polyp segmentation model based on the Segment Anything Model (SAM). We leverage a box-supervised projection loss to confine predicted masks within ground-truth boxes and an augmentation consistency loss to reduce the variability of predicted masks. For adapting SAM, we propose a model architecture AutoPromptSAM, which creates a dense prompt from the image encoder's features for automatic segmentation and employs efficient fine-tuning techniques to transfer the capabilities of the SAM model. Moreover, we distill the model to a lightweight model for quicker inference and less GPU memory cost, which also improves performance. Experimental results on six datasets demonstrate our proposed model and the distilled model show excellent performance in polyp segmentation without relying on any pixel-level labels.