A Deep Learning Multimodal Fusion-Based Method for Cell and Nucleus Segmentation

In recent years, deep learning has been widely utilized in the fields of biomedical image segmentation and cellular image analysis. Supervised deep neural networks trained on annotated data have demonstrated good performance in tasks related to cell and nucleus segmentation. However, the use of supervised models necessitates carefully constructed training data and a substantial amount of ground truth information. Unfortunately, high-quality annotated data for cellular images are scarce. To address the issue of limited datasets, we propose a cell and nucleus segmentation method based on deep learning multimodal fusion. The proposed method includes three modules: a segmentation fundamental module, a multimodal prompter module, and an object output module. This comprehensive approach enables cell and nucleus segmentation tasks to be performed without the need for retraining on new data. The segmentation fundamental module is the core of the framework, as it provides essential segmentation capabilities. By leveraging preexisting models trained on natural imagery, this module effectively performs cell segmentation by incorporating prior knowledge. The multimodal prompter module, a pretrained model, aids in combining image and textual information. It employs a data fusion technique for multiple modalities to deliver prompts that steer the network’s output, thereby avoiding the constraints inherent to single-modality approaches. The object output module combines the inputs from the preceding modules to generate the final segmentation output. The experimental validation confirms the superiority of the proposed method, which outperforms comparative methods in cell and nucleus segmentation tasks and has promise for future applications in cell tracking.