Gaze Assistance for Efficient Segmentation Correction of Medical Images

The segmentation of medical images is an important step in various diagnostic applications, including abnormality detection and radiotherapy planning. Recent developments in Artificial Intelligence (AI) have significantly advanced the field of segmentation automation. However, expert-level accuracy has not been achieved for most segmentation tasks, which significantly hampers the adoption of fully-automated medical image segmentation. This paper investigates the idea of efficient correction of medical image segmentation by using not manual controller commands, which can be time-consuming, but gaze movements. We propose a lightweight fine-tuning approach of the Segment Anything Model in medical images, known as MedSAM, to interactively adjust segmentation masks based on gaze point prompts. Our model is specifically trained for the abdominal CT imaging task using the publicly available WORD database. While surpassing state-of-the-art segmentation models, comprehensive studies with medical experts demonstrated that our gaze-assisted interactive approach led to significant improvements in segmentation quality. Specifically, the gaze-assisted corrections increased the average segmentation performance by nearly 62% for difficult medical cases, compared to traditional segmentation methods based on bounding boxes. The main findings of our proposed work include: 1) the substantial improvement in segmentation quality through gaze interaction, 2) the development of an efficient correction mechanism leveraging eye movements, and 3) the demonstration of gaze-assisted segmentation’s superior performance in abdominal imaging tasks. Our innovative approach shows promise for interactive segmentation of medical images and opens the door for further advancements in human-AI interaction in medicine using eye-tracking technology.