Multi-Scale Co-Attention Transformer Model to Integrate Radiology, Histology, and Genomics: Application to Survival Prediction in Glioblastoma

Considering a "micro" to "macro" view of the tumor, comprehensive clinical evaluation for cancer involves acquiring multi-scale data, including radiology (e.g., CT, MRI) which provides macroscopic morphology and structural tumor details, histology images containing rich phenotypic information at the cellular level, molecular data (e.g., genome sequencing, gene expression, epigenomics, also known as multi-omics) which captures the underlying biological processes, and the clinical data (e.g., age, sex). The ability to comprehensively combine disparate sources of information through computational approaches could enable the discovery of new prognostic markers to reliably predict clinical outcomes. Current deep-learning models combining multiple data modalities have shown promise, but challenges persist in optimizing fusion strategies. This study presents a three-modality, early fusion co-attention transformer (MuSCAT) model, integrating genomics, pathology, and radiology data progressively, to construct integrated prognostic signatures for survival prediction. The integrated prognostic signatures, as demonstrated in survival risk in glioblastoma, achieved the highest C-Index values in 5 fold cross validation using all three modalities (0.651 +/- 0.06) with demonstrated improvement over previous models with early fusion of genomics and pathology (0.549 +/- 0.09).