Search bioRxiv⌕ Search
Home/Archive/bioRxiv

bioRxiv · 10.64898/2026.10.01.752251

VINTER: a generative vision-language model for annotating and interrogating spatial tissues

Wu, J.·Xu, W.·Zhuang, Y.·Zhang, Y.·Loza Lopez, M. d. J.·yang, y.·Nakai, K.
View original source ↗Export metadata ↓

Abstract

Spatial transcriptomics links gene expression to tissue structure, yet current models learn embeddings whose meaning is assigned afterwards or read histology without measured expression. Tissue decisions are therefore rarely made from morphology, genes and neighbourhood together, or examined through them. Here we introduce VINTER, a generative vision-language model aligned on 1.45 million histology-caption pairs that reads histology, measured gene expression and spatial neighbourhood as one sequence. It returns the probability of every tissue class, and because genes enter as text, its molecular evidence can be edited with the image held fixed. Reading all three sources together, VINTER annotated tissue more accurately than spatial-domain methods and a spatial foundation model, and transferred across cohorts without target labels. Editing only the neighbourhood resolved the follicular B-cell and plasma-cell programmes behind its reading of tertiary lymphoid structure (TLS) maturation in lung and kidney cancer. Its class probabilities exposed what single labels conceal in triple-negative breast cancer: immune heterogeneity within stroma and cancer hidden beneath benign labels. Without retraining, the same model transferred to Visium HD and single-cell Xenium data and resolved follicular architecture within colorectal cancer TLS. VINTER thus turns spatial annotation into an interrogable reading of tissue across platforms and resolutions.

Explore related subjects

bioinformaticsPreprint

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Wu, J., Xu, W., Zhuang, Y., Zhang, Y., Loza Lopez, M. d. J., yang, y., Nakai, K.. 2026-10-08. VINTER: a generative vision-language model for annotating and interrogating spatial tissues. https://doi.org/10.64898/2026.10.01.752251

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related preprints

bioRxiv2026

Cross-modal transfer enables zero-shot prediction of drug responses across cell types

Genetic perturbation responses capture the functional consequences of gene-level interventions, providing functional information for drug-response prediction beyond baseline gene expression. Within the same cellular context, genetic and chemical perturbations engage shared regulatory networks, providing a biological basis for transferring response information between perturbation modalities. Here we introduce GRAFT, a framework that combines genetic-to-chemical transfer with cross-context prediction to infer drug-induced transcriptional responses without drug measurements in the target context. Across benchmarked LINCS and external contexts, GRAFT improved response correlation, directional agreement and magnitude reconstruction over competing methods, particularly for drugs with context-dependent responses. GRAFT-Adapt further refines predictions of unmeasured responses using a small number of drug measurements from the target context. GRAFT-Atlas spans 13 cellular contexts and 7,856 drug catalog entries, comprising 102,128 predictions for drug-context combinations. The atlas captured recognizable pharmacological neighborhoods, shared functional programs, context-dependent transcriptional changes and structural-functional associations supported by molecular docking. Across ten iPSC lines from distinct donors, GRAFT identified candidate inter-donor differences, with functional support for selected candidates from measured genetic perturbation responses. Together, GRAFT enables perturbation-informed prediction and comparison of drug responses across cellular contexts.

bioinformatics↗
bioRxiv2026

Interpreting graph neural networks reveals microbial association patterns in inflammatory bowel diseases

Traditional machine learning models trained on gut microbiome abundance profiles can distinguish individuals with inflammatory bowel disease (IBD) from non-IBD individuals. However, their interpretation typically considers taxa in isolation, making it difficult to understand how associations between taxa contribute to predictions. Here, we inferred individual-specific microbial association networks from longitudinal metagenomics data of 106 individuals in HMP2/iHMP and used them to train and interpret graph neural network (GNN) models, which we compared with abundance-based models such as Random Forest (RF). Although RF achieved higher predictive performance (AUPRC ~0.59 vs 0.446), interpreting the GNNs revealed that their predictions relied on a densely connected module of co-abundant Bacteroidales species. Unlike the taxa prioritized by RF, these species were not differentially abundant between conditions. In an independent cohort, this module carried most of the predictive signal, and removing its associations reduced performance to near chance. These results show that coordinated association structure, rather than abundance shifts alone, is a feature of the IBD gut microbiome, and that interpreting GNNs can uncover such patterns that complement abundance-based analyses.

bioinformatics↗
bioRxiv2026

FateDriver: Weakly supervised dual-view graph learning for transition-specific prioritization of candidate regulators in cellular reprogramming

Cellular reprogramming supports regenerative medicine, yet the endogenous regulators of a specified source-to-target transition remain incompletely defined. Existing computational methods rank candidate regulators from network topology, trajectory dynamics, or cell-state labels, leaving unused the factors experimentally delivered to induce the transition. We present FateDriver, a weakly supervised dual-view graph neural network that jointly represents cells and genes from single-cell transcriptomes, treats the delivered factors as sparse gene-level labels, and ranks the remaining genes for a specified transition under a positive--unlabeled formulation. Across two OSKM datasets and one GMT dataset, FateDriver ranked first or second among six methods in recovering externally curated reference genes, exceeded a supervision-matched non-graph model despite weaker co-expression with the delivered factors, and achieved the highest recovery of protein targets of pluripotency-inducing chemical cocktails in both OSKM datasets. Holding either the source or the endpoint fixed yielded distinct candidate programs, including a DDX58--interferon axis associated with cells lacking endogenous pluripotency markers. Applying the same framework to candidate source--target pairs prioritized distinct candidate origins for two in vivo partially reprogrammed intestinal populations. FateDriver converts sparse experimental knowledge into transition-specific hypotheses for regulator prioritization from single-cell trajectories.

bioinformatics↗