Dynamic Detection of Global Micro-Features in Multimodal Retinal Images

Vitreoretinal diseases cause vision disorders and blindness, presenting challenges for surgeons due to operating in dimly lit and complex spaces. Robot-assisted surgery enhances precision and efficiency through high repeatability and advanced imaging. However, preoperative autonomous navigation methods face blurred boundaries, element confusion, and occlusions when detecting intraoperative retinal micro-features. Their core deficiency lies in handling large-scale variations, high local similarity, and insufficient feature extraction during the transition to intraoperative. Therefore, a detection framework for full-vision intraoperative retinal micro-features is established, utilizing multimodal image fusion, dynamic tracking, and feature-sharing, consisting of feature segmentation and matching. A novel TransUNet model with multi-depth partial convolutional residual layers and edge dilation is proposed for intraoperative fundus image segmentation, refining and sharpening micro-features for clear boundary delineation. Subsequently, a multi-scale, two-stage micro-feature matching method fusion of preoperative and intraoperative images accurately completes features in instruments-occluded and invisible areas. A dynamic keypoints tracking and updating mechanism for sequential images overcomes element confusion by inheritance and refreshing contextual information. Experimental results show that the proposed framework can detect retinal micro-features with an accuracy of 90.47%, which is 15.76% higher than intraoperative segmentation. Ideally, it can complement more than 99.70% of features in instruments’ occluded region and more than 97.47% in the invisible region when the visible features area reaches more than 1200 pixels. The proposed framework provides reliable and stable intraoperative reference points for robot localization, significantly enhancing perception and understanding of the environment in robot-assisted vitreoretinal surgeries.