VB-SOLO: Single-Stage Instance Segmentation of Overlapping Epithelial Cells

The instance segmentation of overlapping cells in smear images of epithelial cells is challenging due to the significant overlap and adhesion between the cells’ translucent cytoplasm. In this paper, an improved single-stage instance segmentation network called VoVNet-BiFPN-SOLO (VB-SOLO) is proposed to address this problem. The model takes SOLOv2 model as its main frame. Firstly, the backbone network uses Efficient Channel Attention (ECA) to optimize the VoVNetv2 network to increase the information interaction across channels and enhance the extraction of cell instance features. Secondly, the bi-directional feature pyramid network (BiFPN) is introduced to connect with the new backbone. BiFPN can achieve the weighted fusion of features with different resolutions from bottom to top and keep more shallow semantic information in the network. Finally, the Convolutional Block Attention Module (CBAM) is added to the mask branch to improve cell segmentation results in feature maps. Experimental results on the publicly available datasets CISD and Cx22 demonstrate the effectiveness of the VB-SOLO model, achieving a DCP of 0.966 and 0.940 and a FNRO of 0.055 and 0.03. Compared to the original SOLOv2 algorithm, the proposed method achieved improvements in DCP of 1.3% and 1.1% respectively. Additionally, comparative tests with multiple instance segmentation networks have shown that the proposed improved network can achieve a better balance between segmentation accuracy and efficiency. The experimental results demonstrate the effectiveness of the proposed network improvements and the potential of single-stage instance segmentation networks in overlapping cell image segmentation.