Segmentation of Nuclei in H&E-Stained Histological Images using Deep Learning Framework: A Perspective on Ensemble Approach and Nuclei Count

Quantitative and qualitative analysis of cell nuclei in histopathological images have significant importance in the detection and grading of cancer and other pathologies. In the era of digital pathology and whole slide imaging, automated cell nuclei segmentation is essential for evaluating histopathological images. For that, the deep learning-based approach currently become the popular approach. In the present work, we evaluated the impact of various stain normalization preprocessing and homogenous ensemble deep learning models for cell nuclei segmentation and their counting in hematoxylin & eosin (H&E) stained histopathological images. We used the Squeeze U-Net model for nuclei segmentation and performed comparative analysis on Reinhard, Macenko and Vahadane stain normalization techniques. Reinhard stain normalization technique outperforms Macenko and Vahadane in within- and cross-dataset analysis. The ensemble model approach showed comparable or lower nuclei counting error than without the ensemble approach. We observed that the better dice score, was not always results in better nuclei counting accuracy.