Cell Nuclei Segmentation Using Deep Hybrid Representation Learning for 2D and 3D Microscopy Images

Cell segmentation is a central task in biomedical image analysis. We introduce a 3D deep neural network for 3D cell nuclei segmentation that performs multi-task learning to generate different representations from 3D microscopy images. The network exploits information about cell regions, cell contours, and cell distances. It consists of a 3D Residual U-Net with Squeeze-and-Excitation operations and is trained with a recently introduced loss function that suppresses image regions away from cell borders. To improve cell splitting, we propose a novel representation named signed Instance Distance. Cell instance segmentation is achieved by combining the learned representations to determine cell seeds for a watershed algorithm. We also introduce a 2D version of the network. The method was applied to large-scale 3D micro-CT and 3D electron microscopy datasets from NucMM as well as to 2D fluorescence and phase-contrast microscopy datasets from the Cell Tracking Challenge. For the 3D datasets, the method achieves a large improvement compared to existing methods. For the 2D datasets, the method yields the best segmentation accuracy out of 39 methods for one of the datasets.