Dynamic Detection of Global Microfeatures in Multimodal Retinal Images

Vitreoretinal diseases cause vision disorders and complete vision loss, presenting challenges for surgeons due to operating in dimly lit and complex spaces. Robot-assisted surgery enhances precision (PC) and efficiency through high repeatability and advanced imaging. However, preoperative autonomous navigation methods face blurred boundaries, element confusion, and occlusions when detecting intraoperative retinal microfeatures. Their core deficiency lies in handling large-scale variations, high local similarity, and insufficient feature extraction during the transition to intraoperative. Therefore, a detection framework for full-vision intraoperative retinal microfeatures is established, utilizing multimodal image fusion, dynamic tracking, and feature sharing, consisting of feature segmentation and matching. A novel Transformer-based U-Net (TransUNet) model with multidepth partial convolutional residual layers and edge dilation is proposed for intraoperative fundus image segmentation, refining and sharpening microfeatures for clear boundary delineation. Subsequently, a multiscale, two-stage microfeature matching method fusion of preoperative and intraoperative images accurately completes features in instrument-occluded and invisible areas. A dynamic keypoints’ tracking and updating mechanism for sequential images overcomes element confusion by inheritance and refreshing contextual information. Experimental results show that the proposed framework can detect retinal microfeatures with an accuracy (ACC) of 90.47%, which is 15.76% higher than intraoperative segmentation. Ideally, it can complement more than 99.70% of features in the instruments’ occluded region and more than 97.47% in the invisible region when the visible features’ area reaches more than 1200 pixels. The proposed framework provides reliable and stable intraoperative reference points for robot localization, significantly enhancing perception and understanding of the environment in robot-assisted vitreoretinal surgeries.