CellSeg2TLS: A Deep Learning Framework for Predicting the Maturation of Tertiary Lymphoid Structures in Pathology Images

The maturation of tertiary lymphoid structures (TLSs) is a valuable prognostic factor in cancer immunotherapy. The current method for TLS maturation evaluation is multiplexed immunofluorescence (mIF) staining, which is expensive and thus not yet widely applicable in clinical practice. In this paper, we propose CellSeg2TLS, a novel deep learning framework for predicting TLS maturation from routine hematoxylin and eosin (H&E) stained pathology images. The main idea of CellSeg2TLS is to leverage cell instance segmentation as a proxy task for TLS maturation prediction to mitigate the challenges posed by limited sample size and guide the model to focus on cellular components that are closely related to TLS maturation. To perform the cell segmentation, we design a pipeline to annotate cells automatically by registering the H&E and mIF images of the same tissue. Finally, we propose a cell-tissue feature fusion model to perform TLS maturation prediction, which combines both cell components and topological structure information of the tissues in TLSs. The experiments demonstrate that our method significantly outperforms existing TLS maturation prediction methods and state-of-the-art pathology image classification methods on two datasets. Interpretability analysis indicates that our model is able to capture some tissue structures associated with TLS maturation.