Vision-Based Force Feedback Estimation for Robot-Assisted Surgery Using Instrument-Constrained Biomechanical Three-Dimensional Maps

We present a method for estimating visual and haptic force feedback on robotic surgical systems that currently do not include significant force feedback for the operator. Our approach permits to compute contact forces between instruments and tissues without additional sensors, relying only on endoscopic images acquired by a stereoscopic camera. Using an underlying biomechanical model built on-the-fly from the organ shape and by considering the surgical tool as boundary conditions acting on the surface of the model, contact force can be estimated at the tip of the tool. At the same time, these constraints generate stresses that permit to compose a new endoscopic image as visual feedback for the surgeon. The results are demonstrated on in vivo sequences of a human liver during robotic surgery, whereas quantitative validation is performed on a DejaVu and ex vivo experimentation with ground truth to show the advantage of our approach.