Semantic-aware Graph Reasoning and Guidance for Surgical Scene Understanding

Surgical instrument segmentation is a key technology in enhancing the accuracy and efficiency of robot-assisted surgeries (RAS). This process faces several challenges, including high similarity between different classes of instruments, disturbances caused by varying lighting conditions, and the blurriness resulting from rapid movements. These challenges are intensified when adapting systems without direct access to samples from the specific surgical environments they will be deployed in. To overcome these obstacles, we introduce a Semantic-Guided Adaptive Graph Convolution Network (SGANet), tailored for high-fidelity segmentation within complex surgical scenes. Unlike traditional models that rely on attention mechanisms for feature reweighting—which often fall short in distinguishing between closely related categories—our SGANet employs three innovative components: a Semantic Graph Reasoning Module, Semantic Embedding, and Semantic Guidance. The Semantic Graph Reasoning Module excels in discerning and linking various semantic features, harnessing long-range spatial relationships to differentiate and dynamically adapt its graph structure according to the input data. Meanwhile, the Semantic Embedding and Guidance components utilize these nuanced semantic connections to direct the reconstruction of high-resolution features, promoting clarity within classes and distinction between them. Our method’s effectiveness is validated through rigorous testing on the public datasets EndoVis 2017 and EndoVis 2018, where it outperforms current leading methods across different backbone architectures.