A Fused Transformer-based Model for Gene Expression Prediction using Histopathology Images

Spatial transcriptomics (ST) is a cutting-edge technology that enables the spatial localization and analysis of gene expression within tissue sections. Despite its transformative potential, ST is constrained by high costs and limited spatial resolution, making it less accessible and challenging to implement widely. To overcome these limitations, we propose a fused transformer-based approach designed to predict high-density gene expression profiles directly from Whole Slide Images (WSIs). Our model capitalizes on the multi-scale hierarchical structure of WSIs, integrating several key components: ResNet50 and transformer encoders to extract features at various scales, and a Fused Transformer Block (FTB) that effectively aggregates these features. The FTB incorporates Spatial Positional Embeddings (SPE) to maintain spatial context, cross attention to prioritize relevant features across different scales, and Random Mask Attention (RMA) to concentrate the model’s attention on the most significant patterns. To validate our approach, we conducted extensive experiments using three public spatial transcriptomics datasets (HBC, HER2+, SCC) and three additional Visium datasets from 10X Genomics. Our results demonstrate that the proposed model not only preserves essential spatial relationships for mapping gene expressions to tissue morphology but also outperforms current state-of-the-art methods.