Perreard, L. | Search bioRxiv

bioRxiv2023

Spatial Omics Driven Crossmodal Pretraining Applied to Graph-based Deep Learning for Cancer Pathology Analysis

Graph-based deep learning has shown great promise in cancer histopathology image analysis by contextualizing complex morphology and structure across whole slide images to make high quality downstream outcome predictions (ex: prognostication). These methods rely on informative representations (i.e., embeddings) of image patches comprising larger slides, which are used as node attributes in slide graphs. Spatial omics data, including spatial transcriptomics, is a novel paradigm offering a wealth of detailed information. Pairing this data with corresponding histological imaging localized at 50-micron resolution, may facilitate the development of algorithms which better appreciate the morphological and molecular underpinnings of carcinogenesis. Here, we explore the utility of leveraging spatial transcriptomics data with a contrastive crossmodal pretraining mechanism to generate deep learning models that can extract molecular and histological information for graph-based learning tasks. Performance on cancer staging, lymph node metastasis prediction, survival prediction, and tissue clustering analyses indicate that the proposed methods bring improvement to graph based deep learning models for histopathological slides compared to leveraging histological information from existing schemes, demonstrating the promise of mining spatial omics data to enhance deep learning for pathology workflows.

pathology↗

bioRxiv2022

Inferring Spatially Resolved Transcriptomics Data from Whole Slide Images for the Assessment of Colorectal Tumor Metastasis: A Feasibility Study

Over 150,000 Americans are diagnosed with colorectal cancer (CRC) every year, and annually over 50,000 individuals will die from CRC, necessitating improvements in screening, prognostication, disease management, and therapeutic options. Tumor metastasis is the primary factor related to the risk of recurrence and mortality. Yet, screening for nodal and distant metastasis is costly, and invasive and incomplete resection may hamper adequate assessment. Signatures of the tumor-immune microenvironment (TIME) at the primary site can provide valuable insights into the aggressiveness of the tumor and the effectiveness of various treatment options. Spatially-resolved transcriptomics technologies offer an unprecedented characterization of TIME through high multiplexing, yet their scope is constrained by cost. Meanwhile, it has long been suspected that histological, cytological and macroarchitectural tissue characteristics correlate well with molecular information (e.g., gene expression). Thus, a method for predicting transcriptomics data through inference of RNA patterns from whole slide images (WSI) is a key step in studying metastasis at scale. In this work, we collected and preprocessed Visium spatial transcriptomics data (17,943 genes at up to 5,000 spots per patient sampled in a honeycomb pattern) from tissue across four stage-III matched colorectal cancer patients. We compare and prototype several convolutional, Transformer, and graph convolutional neural networks to predict spatial RNA patterns under the hypothesis that the transformer and graph-based approaches better capture relevant spatial tissue architecture. We further analyzed the models ability to recapitulate spatial autocorrelation statistics using SPARK and SpatialDE. Overall, results indicate that the transformer and graph-based approaches were unable to outperform the convolutional neural network architecture, though they exhibited optimal performance for relevant disease-associated genes. Initial findings suggest that different neural networks that operate on different scales are relevant for capturing distinct disease pathways (e.g., epithelial to mesenchymal transition). We add further evidence that deep learning models can accurately predict gene expression in whole slide images and comment on understudied factors which may increase its external applicability (e.g., tissue context). Our preliminary work will motivate further investigation of inference for molecular patterns from whole slide images as metastasis predictors and in other applications.

pathology↗