Roy, T. K. | Search bioRxiv

bioRxiv2025

Query-driven generative AI synthesizes multi-modal spatial omics from histology

Spatial omics technologies offer unprecedented insights into the cellular organization of tissues; however, they are not yet scalable for routine clinical use. In contrast, images of histological staining remain the foundation of pathological diagnosis despite lacking molecular information. Bridging this gap requires computational methods that can accurately infer spatial molecular data from histology alone. Here, we introduce TissueCraftAI, a generative artificial intelligence framework that predicts multi-modal spatial omics maps directly from standard histology images using natural language prompts. To train and validate our model, we created PRISM-12M, a large-scale dataset comprising over twelve million spatially registered histology and spatial omics image patches across fourteen tissue types from humans and mice. TissueCraftAI significantly outperforms existing methods in generating realistic histology images and predicting spatial proteomics and transcriptomics data with high fidelity. We demonstrated its utility in various downstream applications, including improving cell type annotation and enhancing the accuracy of patient survival predictions across multiple cancer types. By enabling flexible, query-driven in silico spatial molecular analysis using routine histology images, TissueCraftAI opens up new research avenues in computational pathology.

genomics↗

bioRxiv2024

CelloType: A Unified Model for Segmentation and Classification of Tissue Images

Cell segmentation and classification are critical tasks in spatial omics data analysis. We introduce CelloType, an end-to-end model designed for cell segmentation and classification of biomedical microscopy images. Unlike the traditional two-stage approach of segmentation followed by classification, CelloType adopts a multi-task learning approach that connects the segmentation and classification tasks and simultaneously boost the performance of both tasks. CelloType leverages Transformer-based deep learning techniques for enhanced accuracy of object detection, segmentation, and classification. It outperforms existing segmentation methods using ground-truths from public databases. In terms of classification, CelloType outperforms a baseline model comprised of state-of-the-art methods for individual tasks. Using multiplexed tissue images, we further demonstrate the utility of CelloType for multi-scale segmentation and classification of both cellular and non-cellular elements in a tissue. The enhanced accuracy and multi-task-learning ability of CelloType facilitate automated annotation of rapidly growing spatial omics data.

bioinformatics↗