Multi-scope Analysis Driven Hierarchical Graph Transformer for Whole Slide Image Based Cancer Survival Prediction

Abstract

<p>Cancer survival prediction requires considering not only the biological morphology but also the contextual interactions of tumor and surrounding tissues. The major limitation of previous learning frameworks for whole slide image (WSI) based survival prediction is that the contextual interactions of pathological components (<em>e.g.</em>, tumor, stroma, lymphocyte, <em>etc.</em>) lack sufficient representation and quantification. In this paper, we proposed a multi-scope analysis driven Hierarchical Graph Transformer (HGT) to overcome this limitation. Specifically, we first utilize a multi-scope analysis strategy, which leverages an in-slide superpixel and a cross-slide clustering, to mine the spatial and semantic priors of WSIs. Furthermore, based on the extracted spatial prior, a hierarchical graph convolutional network is proposed to progressively learn the topological features of the variant microenvironments ranging from patch-level to tissue-level. In addition, guided by the identified semantic prior, tissue-level features are further aggregated to represent the meaningful pathological components, whose contextual interactions are established and quantified by the designed Transformer-based prediction head. We evaluated the proposed framework on our collected Colorectal Cancer (CRC) cohort and two public cancer cohorts from the TCGA project, <em>i.e.</em>, Liver Hepatocellular Carcinoma (LIHC) and Kidney Clear Cell Carcinoma (KIRC). Experimental results demonstrate that our proposed method yields superior performance and richer interpretability compared to the state-of-the-art approaches.<br></p>