Color Space-based HoVer-Net for Nuclei Instance Segmentation and Classification

Nuclei segmentation and classification is a crucial step utilized throughout various microscopy medical analysis applications. However, it has several challenges such as the segmentation of small objects, highly imbalanced data, and subtle differences between types of nuclei. In this paper, we propose a method to tackle the aforementioned problems. First, a new encoder “ConvNeXt” is proposed for the HoVer-Net architecture to leverage the major components of vision transformer to improve the network's encoding capabilities. Second, to better distinguish the differences between nuclei, a multi-channel color space-based approach is proposed to aid the model in extracting discriminative features. Third, affine transformations such as scaling and shearing are utilized to improve the model's gen-eralizability. Finally, Unified Focal Loss (UFL) and Sharpness-Aware Minimization (SAM) are tested in attempt to tackle the foreground-backward imbalance and further improve gen-eralizability, respectively. We show how our proposed method (ConvNeXt-S encoder + affine transformation + multi-channel color stacking) outperforms the original HoVer-Net network on the “Lizard” dataset by 2.6% mPQ+.