M3 -DEGREES Net: Monocular-guided Metric Marching Depth Estimation with Graph-based Relevance Ensemble for Endoluminal Surgery

Robotic endoluminal surgery has gained tremendous attention for its enhanced treatments in gas-trointestinal intervention, where navigating surgeons with monocular camera-based metric depth estimation is a vital sector. However, existing methods either rely on external sensors or perform poorly in terms of visual navigation. In this work, we present our M3 -Degrees Net, a novel monocular vision-guided and graph learning-based network tailored for accurate metric marching depth (MD) estimation. We first leverage a generative model to output a scale-free depth map, providing a depth basis in a coarse granularity. To achieve an optimized and metric MD prediction, a relational graph convolutional network with multi-modal visual knowledge fusion is devised. It utilizes shared salient features between keyframes and encodes their pixel differences on the depth basis as the main node, while a projection length-based node that predicts the MD on a proportional relationship basis is introduced, aiming to enable the network with explicit depth awareness. Moreover, to compensate for rotation-induced MD estimation bias, we model the endoscope's orientation changes as image-level feature shifts, formulating an ego-motion correction node for MD optimization. Lastly, a multi-layer regression network for the metric MD estimation with finer granularity is devised. We validate our network on both public and in-house datasets, and the quantitative results reveal that it can limit the overall MD error under 27.3%, which vastly outperforms the existing methods. Besides, our M3 -Degrees Net is qualitatively tested on the in-house clinical gastrointestinal endoscopy data, demonstrating its satisfactory performance even under cavity mucus with varying reflections, indicating promising clinical potentials.