Improving Cell Nuclei Segmentation in Pathological Tissues Using Self-Supervised Regression Method

Digital pathology, integral to cancer diagnosis, heavily utilizes the analysis of whole slide images (WSIs) to detect cancer cells. Traditionally, WSIs are manually examined—a process that is not only time-consuming but also subject to observer variability. This analysis typically focuses on the assessment of nuclei shape and size, crucial indicators of cancer stage and progression. Given these challenges, accurate nuclei segmentation is pivotal for aiding pathologists in cancer prognosis and staging, as well as for enhancing automated systems designed to assess similar aspects. Recent advances in deep learning, specifically Convolutional Neural Networks (CNNs), have proven effective in processing digital pathology data. Yet, CNNs generally require extensive labeled datasets, often unavailable in medical imaging. This study introduces a novel self-supervised learning approach for nuclei segmentation in WSIs, comparing its efficacy against other self-supervised techniques and traditional methods. It incorporates various self-supervised strategies like image-scale regression and classification, denoising autoencoder, and relative positioning, with a focus on a scale-regression-based approach that significantly outperforms others. Tested on two publicly available histopathological datasets, our method showed substantial improvements in nuclei segmentation, notably outperforming the second-best method by 9.2% on the TNBC dataset and achieving competitive results on the MoNuSeg dataset, with Aggregated Jaccard Indexes (AJI) of 0.73 and 0.619, and F1 scores of 0.927 and 0.804, respectively. These findings underscore the potential of self-supervised learning, especially via scale regression, to enhance the accuracy of training models on WSI images.