Rethinking MedSAM: Performance Discrepancies in Clinical Applications

Segment Anything Model (SAM) has gained significant attention for its versatility and effectiveness in solving various image segmentation problems. Medical image segmentation (MIS) is a complicated problem compared to natural image segmentation, considering the variability in imaging modalities, applications, and clinical requirements. We conduct an evaluation study to validate the usage of MedSAM as a foundation model for the segmentation of medical images, examining its performance in both zero-shot and fine-tuned settings. This paper critically evaluates MedSAM's performance across diverse clinical applications, highlighting performance gaps that question its generalization ability in MIS. Experiments on multiple datasets, including MRI, CT, ultrasound, endoscopy, and retinal fundus images, have shown that MedSAM struggles with complex anatomical structures. Furthermore, the results show that it is sensitive to the bounding box accuracy. These findings suggest that its current implementation lacks the robustness necessary for widespread adoption in clinical practice without fine-tuning. This paper discusses these limitations and proposes strategies for enhancing the MedSAM to address the complexities of MIS.