PASCL: Pathway-based Sparse Deep Clustering for Identifying Unknown Cancer Subtypes

Cancer is a heterogeneous disease which has several subtypes that can be distinguished by molecular, histopathological, and clinical stages. Accurate diagnosis of cancer subtypes is vital to identify distinct disease states and develop effective personalized therapies. A number of unsupervised machine learning techniques have been applied to genomic data of the tumor samples, where clusters of patients were formed to be associated with a clinical outcome such as the survival of patients. However, clustering methods based on distance (or similarity) between data often fail to cluster biological data, due to their nonlinearity. In this paper, we develop a PAthway-based Sparse deep CLustering (PASCL) method for the identification of cancer subtypes. PASCL incorporates prior biological knowledge from pathway databases to build a robust and biological interpretable model. We evaluated the performance of PASCL by comparing with several state-of-the-art clustering methods. PASCL outperformed the benchmarking methods with lowest p-value in logrank test, and its outstanding performance is statistically assessed. PASCL provides a solution not only to effectively identify subtypes using high-dimensional nonlinear genomic data, but also to biologically interpret the model at a pathway level.