NeRF-based multi-frame 3D integration for 3D endoscopy using active stereo

3D measurement for endoscopic systems has large potential not only for cancer diagnosis or computer-assisted medical systems, but also for providing ground truth for supervised training of deep neural networks. To achieve it, one of the promising approach is the implementation of an active-stereo system using a micro-sized pattern-projector attached to the head of the endoscope. Furthermore, a multi-frame optimization algorithm for the endoscopic active-stereo system has been proposed to improve accuracy and robustness; in the approach, differential rendering algorithm is used to simultaneously optimize the 3D scene represented by triangle meshes and the camera/projector poses. One issue with the approach is its dependency on the accuracy of the initial 3D triangle mesh, however, it is not an easy task to achieve sufficient accuracy for actual endoscopic systems, which reduces the practicality of the algorithm. In this paper, we adapt neural radiance field (NeRF) based 3D scene representation to integrate multi-frame data captured by active-stereo system, where the 3D scene as well as the camera/projector poses are simultaneously optimized without using the initial shape. In the experiment, the proposed method is evaluated by performing 3D reconstruction using both synthetic and real images obtained by a consumer endoscopic camera attached with a micro-pattern-projector.Clinical relevance— One-shot endoscopic measurement of depth information is a practical solution for cancer diagnosis, computer-assisted interventions, and making annotations for machine learning training data.