Fast 3D SAM: Crafting Large Vision Models for 3D Medical Image Segmentation via Token Clustering and Fine-Tuning

The Segment Anything Model (SAM) [1] revolutionizes image segmentation, introducing a groundbreaking approach. Despite the promising techniques demonstrated in previous studies for developing 3D variants of SAM with additional modules, the high training parameters and computational burden impede real-time demanding applications such as interactive segmentation. To this end, we present Fast 3D SAM, a novel and lightweight 3D SAM network that integrates parameter-efficient fine-tuning and token clustering, effectively addressing the challenges of heavy computational requirements. Specifically, we propose Efficient Hyper-adapter Module and 3D Token Clustering techniques, which significantly reduce parameter count and alleviate calculation overload. Remarkably, our method outperforms previous state-of-the-art fine-tuning approaches, while requiring only 60% of the parameters and 50% of the Multiply-Accumulate (MACs) operations compared with them.