Nuclei Segmentation in ER-IHC Stained Histopathology Images using Mask R-CNN

Breast cancer is the leading cause of mortality among women in both developing and underdeveloped countries. The nuclei segmentation in digital histopathology image analysis plays a crucial role in breast cancer in the early stages of its development and may allow patients to have proper treatment. Nuclei overlap and complex structural organisation of the breast tissue in biopsy images make nuclei segmentation and feature extraction challenging. To mitigate the aforementioned problems, this paper employed a mask region-based convolution neural network (Mask R-CNN) to segment immunohistochemistry breast cancer images. The mask R-CNN algorithm introduces advanced Regional Proposal Network architecture that precisely addresses the object location to generate candidate regions. The Mask R-CNN used resnet50 as the backbone and applied Feature Pyramid Network (FPN) to fully explore multiscale feature maps. And then, Region Proposal Network (RPN) was used to propose candidate bounding boxes. The robustness of the Mask R-CNN model is enhanced by training the model with our collected dataset. The proposed architecture has the average of 72% precision, 84.2% recall, 77.62% F1-score, and Jaccard Index overall score of 0.59. The proposed model can be beneficial in assisting pathologist for a routine exam, as well as a second opinion for breast cancer segmentation from whole slide images. Since the process is fully automated, it can be done without supervision and only the final result will be attended by the pathologists.