Semantic and Instance Segmentation of Multi-organ Cell Nuclei Using Deep Learning Based Methods

In this paper, a semantic and instance-based segmentation study for cell nuclei in multi-organ histology images is presented. In the proposed study, it was aimed to segment the cell nuclei in the Multi-organ Nucleus Segmentation (MoNuSeg) Challenge 2018 dataset by using different deep learning-based methods such as UNet, TransUNet, Swin-Unet, Mask R-CNN and YOLOv7. Nuclei segmentation performances of the related methods were analyzed and reported for two separate segmentation tasks: semantic segmentation and instance-based segmentation. Experimental studies show that the UNet model is the most successful model in the semantic segmentation task with 0.8814 DSC (Dice Similarity Coefficient) and 0.7946 IoU (Intersection over Union) values, and in the instance segmentation task with 0.5700 PQ (Panoptic Quality) score. The proposed study compares the performance of deep learning-based models that can be used for cell nucleus segmentation, reveals the relationship between the semantic and instance-based segmentation, and discusses the factors affecting performance. The findings are guiding for future model research for cell nuclei segmentation tasks.