FSAM: Fine-tuning SAM encoder and decoder for Medical Image Segmentation

Recently, the Segment Anything Model (SAM), a large pre-training model, has achieved excellent results on natural image segmentation tasks and received extensive attention. However, SAM on medical image segmentation is unsatisfactory since there are significant differences between natural and medical images. How to extend the SAM’s powerful segmentation capabilities to the medical domain requires further exploration. To this end, we propose a simple and efficient fine-tuning approach for SAM that does not require large-scale data called FSAM. Specifically, FSAM simultaneously fine-tunes the encoder and decoder while freezing the prompt encoder. This allows the encoder to extract medical image features effectively and guide decoder segmentation predictions. FSAM achieves state-of-the-art results on eight public medical image datasets, outperforming SAM by +15.39%. Moreover, FSAM exhibits weaker sample scale dependence. The proposed framework further improves the segmentation capabilities of SAM in medical images.