Automated Real-time 3D Visual Servoing Control of Single Cell Surgery Processes

In this article, we propose an automated real-time 3D visual servoing control of single cell surgery processes, which integrates the automation of micromanipulator calibration process, cell initial setup process, and the visual serving control surgery operation. During the cell initial setup process, a deep learning-based object detection algorithm is developed to compute the cell location in real-time for cell surgery. These coordinates are used as feedback derived two-dimensional (2D) images acquired under low microscope magnification for the motion control of a motorized microscope stage. Visual serving control of cell surgery process is achieved through a cell blastomere centroid three-dimensional (3D) localization algorithm. This algorithm computes in real-time centroid position feedback of the blastomere of interest in the 3D image coordinate from 3D Z-stack images. As the most important function within the overall 3D localization algorithm, a 3D edge detection algorithm is used to extract the edges inside the blastomere. The extracted edges are converted into 3D point cloud which is used to determine the centroid location through 3D K-mean clustering algorithm. A position based visual servoing (PBVS) controller is then used to maneuver the micropipette under closed-loop control enabling various cell surgery procedures to be carried out.