An Attention Based Model for Histopathology Image Nuclei Segmentation

In histopathology image analysis, accurate segmentation of nuclei holds immense significance, particularly in the early detection and treatment of diseases like breast cancer. Nuclei segmentation is a fundamental but challenging task due to the intricate variations in nuclear shapes, sizes, densities, and overlapping instances. In this paper, we propose a segmentation model based on CNNs and transformers with attention layers. This experiment aims to demonstrate that the proposed deep learning model performs better on two different types of datasets: immunohistochemistry (IHC) stain and hematoxylin and eosin (H&E) stain images. To evaluate the model's performance, we use a privately collected IHC dataset and a public dataset called MoNuSeg for H&E images. The proposed architecture achieves an average precision of 0.8792, recall of 0.8676, F1-score of 0.8734, and a IoU overall score of 0.7456 for the IHC dataset, while for the H&E dataset, it achieves a precision of 0.8974, recall of 0.8736, F1-score of 0.8848, and a IoU overall score of 0.7245. This research will serve as a foundation for the future development of more complex deep learning models, potentially incorporating cascades or combinations of the models studied.