Overlapping Cell Nuclei Segmentation in Digital Histology Images using Intensity-based Contours

Automated nuclei segmentation techniques in histopathological image analysis continue to improve. The machine learning model requires the annotation of large data sets which is a time-consuming, expensive, and laborious process. This segmentation is also limited in detecting touching or overlapping nuclei and considers any overlapping nuclei as a single nucleus. This is due to low contrast images, occultation, and diversity of cell nuclei. This work proposes an automated overlapping nuclei segmentation model with a U-net and an intensity-based contour technique in order to address these issues. In a previous study, a U-net segmentation model was trained with synthetic data, which was generated using a GAN model, where a small number of histopathology data was used to generate the synthetic data. This reduced the data limitation and need for nuclei annotation in the deep learning model. Initially in this study, the overlapping nuclei regions were not considered for segmentation by the network. Hence, an intensity-based contour line is proposed to separate overlapping nuclei regions. The distance transformation is utilized to define the center of each nucleus. The identification of local minima followed by intensity-based gradient weights is applied to obtain the final segmentation line of overlapping nuclei. The boundary of the overlapping nuclei is refined, and noise is removed in order to clearly describe each nuclei region. The proposed method results in 91.6% accuracy in separating the overlapping nuclei compared to other existing methods.