Nuclei Segmentation in Histopathology Images Using Rotation Equivariant and Multi-level Feature Aggregation Neural Network

The histopathological analysis is the gold standard for assessing the presence and many complex diseases, like tumors. As one of the essential part of tumors, the shape, staining, and tissue distribution of the nuclei plays an important role in tumor diagnosis. However, due to nuclei congestion and possible occlusion, nuclei segmentation remains challenging. In this paper, we propose an automatic and effective nuclei segmentation method in histopathology images based on rotation equivariant and multi-level feature aggregation neural network (REMFANet). First, considering the inherent rotation equivariant of digital pathological images, we introduce group equivariant convolutions to improve the performance of the automatic segmentation of pathological images. Second, to eliminate the semantic gap between shallow features and deep features in encoder-decoder structural models, we propose a multi-level feature aggregation strategy based on U-Net 3+. Specifically, (1) we design a new decoder module to restore pixel-level predictions more accurately; (2) we propose an improved long-skip connection mode to provide richer semantic information in the decoder; (3) we also construct a semantic enhancement block to enhance the robustness of lowlevel semantic information. Finally, we evaluate our REMFA-Net on the MoNuSeg dataset and compare the results with seven state-of-the-art methods. Experimental results demonstrate the superiority of the proposed method over other models for the nuclei segmentation in histopathology images.