Automated Live Cell Evaluation via a CNN-Transformer Combined Microscopy Image Enhancement Network

Automated morphological measurement of cellular and subcellular structures in live cells is important for evaluating cell functions. Due to their small size and transparent appearance, visualizing cellular and subcellular structures often requires high magnification microscopy and fluorescent staining. However, high magnification microscopy gives a limited field of view, and fluorescent staining alters cell viability and/or activity. Therefore, microscopy image enhancement methods have been developed to predict detailed intracellular structures in live cells. Existing image enhancement networks are mostly CNN-based models lacking global information or Transformer-based models lacking local information. For these purposes, a novel CNN-Transformer combined bilateral U-Net (CTBUnet) is proposed to effectively aggregate both local and global information. Experiments on the collected sperm cell enhancement dataset demonstrate the effectiveness of proposed network for both super-resolution and virtual staining prediction. Note to Practitioners—Automated and accurate intracellular morphology measurement is crucial for cell quality analysis. Microscopy image enhancement methods including super-resolution and virtual staining prediction were proposed to enhance or highlight details of subcellular structures without high magnification microscopy or invasive staining. To effectively combine local and global information, a novel CNN-Transformer combined image enhancement network is proposed. Different from traditional CNN-Transformer combined structures that only directionally fuse outputs from CNN and Transformer, the proposed bilateral fusion module bidirectionally fuses and exchanges features from CNN and Transformer. The proposed bilateral fusion module incorporates not only channel-wise fusion but also spatial-wise fusion to effectively aggregate local and global features. Additionally, a region-aware attention gate is proposed to urge the network to only focus on reconstructing cell structures regardless of background. The proposed method outperformed existing networks with a better enhancement effect for subcellular details.