Prompt Engineering in Medical Image Segmentation: An Overview of the Paradigm Shift

Foundation AI models have emerged as powerful pre-trained models on a large scale, capable of seamlessly handling diverse tasks across multiple domains with minimal or no fine-tuning. These models, exemplified by the impressive achievements of GPT-3 and BERT in natural language processing (NLP), as well as CLIP and DALL-E in computer vision, have garnered considerable attention for their exceptional performance. A noteworthy addition to the realm of image segmentation is the Segment Anything Model (SAM), a foundation AI model that revolutionizes image segmentation. With a single click or a natural language prompt, SAM exhibits the remarkable ability to segment any object within an image, marking a significant paradigm shift in medical image segmentation. Unlike conventional approaches that rely on labeled data and domain-specific knowledge, SAM breaks free from these constraints. Deep convolutional neural network (DCNN)-based, SAM comprises an image encoder, a prompt encoder, and a mask decoder, showcasing its efficient and flexible architecture. Medical image segmentation, in particular, benefits from SAM’s exceptional speed and high-quality segmentation. In this paper, we delve into the effectiveness of SAM for medical image segmentation shedding light on its capabilities. Moreover, our investigation explores the strengths and limitations of prompt engineering in medical computer vision applications, not only encompassing SAM but also other foundation AI models. Through this exploration, we unravel their immense potential in catalyzing a paradigm shift in the field of medical imaging.