Search bioRxiv⌕ Search

bioRxiv · 10.1101/2023.07.11.548593

Distinguishing word identity and sequence context in DNA language models

Abstract

Transformer-based large language models (LLMs) are very suited for biological sequence data, because the structure of protein and nucleic acid sequences show many analogies to natural language. Complex relationships in biological sequence can be learned, although there may not be a clear concept of words, because they can be generated through tokenization. Training is subsequently performed for masked token prediction. With this strategy, the models learn both the token sequence identity and a larger sequence context. We developed a framework to interrogate what a model learns, which is both relevant for the interpretability of the model and to evaluate its potential for specific tasks. We used a DNA language model, which is trained on the human reference genome with a Bidirectional Encoder Representations from Transformers (BERT) model. In this model, tokens are defined with overlapping k-mers. To gain insight into the models learning, we interrogated how the model performs predictions, extracted token embeddings, and defined a fine-tuning benchmarking task to predict the next tokens of different sizes without overlaps. This task is very suited to evaluate different pretrained DNA language models, also called foundation models, since it does not interrogate specific genome biology, does not depend on the tokenization strategy, the size of the vocabulary, the dictionary, or the number of parameters used to train the model. Lastly, the task performs without leakage of information from token identity into the prediction task, which makes it particularly useful to evaluate the learning of sequence context. Through this assessment we discovered that the model with overlapping k-mers struggles to learn larger sequence context. Instead, the learned embeddings largely represent token sequence. Still, good performance is achieved for genome biology inspired fine-tuning tasks. Models with overlapping tokens may be used for tasks where a larger sequence context is of less relevance, but the token sequence directly represents the desired learning feature. This emphasizes the need to interrogate knowledge representation in biological large language models. Transparency is particularly important for biomedical use cases and an understanding of what the models are learning can be used to match the model to the desired task.

Source connections

Explore related subjects

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Sanabria, M., Hirsch, J., Poetsch, A. R.. 2023-07-12. Distinguishing word identity and sequence context in DNA language models. https://doi.org/10.1101/2023.07.11.548593

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related preprints

Machine Learning-Enhanced Mucosal Luminal Interface Metaproteomics Reveals Diagnostic Potential of Microbial Proteins in Pediatric Inflammatory Bowel Disease

Abstract: Background: Inflammatory Bowel Disease (IBD), a chronic gastrointestinal disorder with two main subtypes, Crohn's Disease (CD) and Ulcerative Colitis (UC), is associated with more aggressive phenotypes and greater diagnostic complexity in pediatric patients. Conventional diagnostic biomarkers, lack sufficient distinguishing power between IBD subtypes. This study aimed to investigate and compare proteomic and metaproteomic profiles of pediatric IBD (PIBD) patients and healthy controls to develop a diagnostic panel capable of accurately diagnose IBD and distinguish its subtypes. Methods: We performed an in-depth evaluation of mucosal luminal interface samples from PIBD patients and healthy individuals, separately analyzing both the human proteome and microbial metaproteome data. The study involved selecting appropriate protein sets, employing feature selection techniques and machine learning across three intestinal regions: the terminal ileum, ascending colon, and descending colon. Results: Notably, panels composed solely of microbial biomarkers outperformed both human-only and combined human-microbial panels, achieving highly accurate results. Furthermore, the ascending colon emerged as the optimal sampling site, a result that aligns with the rich microbial diversity revealed in its taxonomic profile. Conclusions: Using a machine learning-based strategy on combined proteomics and metaproteomics, this study uncovered disease-relevant signaling pathways and delivered taxonomic as well as functional insights into the microbial dysbiosis of PIBD. A diagnostic panel built around microbial proteins (particularly those originating from the ascending colon) emerges as a promising tool for more precise stratification of PIBD subtypes. Larger, prospective studies are now needed to validate these candidate biomarkers and translate them into clinically deployable diagnostic panels. Keywords: Metaproteome, Biomarker, Pediatric Inflammatory bowel disease, Machine learning, Mucosal luminal interface.

bioinformatics↗

Revealing the transcriptomic organization of extrasomatic interfaces using SCOUBI

Spatial transcriptomics (ST) generates substantial transcript signal outside segmented cell bodies, which is generally discarded by analysis tools. This is especially problematic for brain ST data, as neuronal mRNAs can be localized far from somata - in axons, dendrites, and synaptic compartments - and up to half of the transcripts in ST data may lie outside cell bodies. Here, we present SCOUBI, a computational framework that identifies axon-dendrite interfaces from extrasomatic RNA in ST data, revealing a hitherto overlooked layer of molecular function in neural tissues. It first partitions extrasomatic space into axon- and dendrite-enriched compartments based on known marker genes, using a neural network trained to maximize marker specificity and inter-compartment communication signals. It then defines interfaces as loci where predicted axonal and dendritic compartments converge and exhibit ligand-receptor colocalization. Applied to mouse brain, SCOUBI creates maps of axonal and dendritic compartments and their interfaces that align with known spatial patterns of neurite and synaptic distribution. It identifies transcriptomic differences between interfaces and the surrounding extrasomatic space, revealing region-specific signatures of neuronal signaling and synaptic plasticity. By comparing interface transcript enrichment with local somatic expression, SCOUBI identifies genes that preferentially localize at sites of intercellular communication rather than simply reflecting nearby cell bodies. SCOUBI reveals interface-associated transcripts beyond established synaptic genes, with molecular signatures reproducible across independent samples and imaging platforms. In summary, SCOUBI transforms a substantial and often-discarded aspect of ST data into an interpretable readout of subcellular organization and neural signaling.

bioinformatics↗

Joint Analysis of Common Regulatory and Rare Coding Variants for Improved Power in Kernel Association Tests

Rare alleles arise from mutations on an existing haplotype background, yet current gene-based rare variant association tests treat low-frequency coding variants as effects independent of the common regulatory variants that surround them. The penetrance of rare coding variants can be modulated by the cis-regulatory landscape, and failing to account for this context limits the statistical power of existing methods for complex traits. We present PrediXSKAT, a kernel-based rare variant association framework that incorporates the surrounding cis-regulatory landscape by estimating its impact on gene expression via PrediXcan. PrediXSKAT embeds predicted gene expression directly into the Sequence Kernel Association Test kernel through element-wise similarity weighting, concentrating rare variant signal among individuals who share concordant genetic and regulatory backgrounds. Comprehensive simulations demonstrate that PrediXSKAT increases power over standard kernel-based testing under biologically realistic conditions where a proportion of tested variants carry small, mixed-direction effects. Type I error rates are controlled in the absence of association. Power gains persist across a range of sample sizes and causal variant architectures. Applied to whole-genome sequencing data from the Alzheimer's Disease Sequencing Project, PrediXSKAT recapitulates established disease loci and amplifies signals at previously unreported candidate genes without indiscriminately inflating genome-wide results. PrediXSKAT provides a generalizable and extensible framework for the joint analysis of common regulatory and rare coding variants. The approach can be applied to any kernel-based set test and extended to jointly consider multiple genes within the same biological pathway, offering a principled strategy for improving rare variant discovery in large-scale sequencing studies of complex disease.

bioinformatics↗