Under which circumstances do genomic neural networks learn motifs and their interactions?

Abstract

A central challenge in biology is to understand, predict, and engineer the second genetic code: how sequence encodes gene expression. Two components of this challenge are: (1) accurate prediction (and design) of gene expression from sequence and (2) mechanistic understanding of how sequence-to-expression encoding actually works in cells. A powerful general approach to this problem is to combine large scale data generation with artificial intelligence. For example, massively parallel reporter assays (MPRAs) can quantify the expression of thousands of different sequences in pooled experiments and the resulting data can be used to train deep learning models. Unlike in the case of long-context genomic language models, where transformer-based architectures are a dominant paradigm, it remains contested whether for MPRA datasets other architectural components can lead to more useful, generalizable predictors, and whether they affect model interpretability, i.e. the ability to capture causal biological mechanisms (either inherently or when using downstream interpretability or explainability techniques, "xAI"). Ablation analyses may help elucidate important architectural components, but are almost always anecdotal, unable to describe generalizable tendencies, as they are done with a single training dataset or a few testing datasets. Here, we attempt to reconcile concerns and provide guidance for MPRA model design and xAI choice by simulating at scale 1,500 motif-based genetic architectures and evaluating the ability of different model architecture-xAI pairs to first predict an outcome given a sequence as input, and second, report involved motifs and their corresponding grammar. We find that attention-based models are efficient learners, and while we recommend their use in low-data regimes, their performance is surpassed by alternative models, like dilated CNNs, under larger sample sizes. We next show that across grammars and models, current methods for motif extraction converge toward reporting the same set of motifs, which is dominated by motifs with large effect sizes. We then perform in silico experiments across models and their discovered motifs and find that these methods accurately rank motifs based on learned effect size, but that their learned effect size is systematically miscalibrated, particularly in the presence of interactions (epistasis). Finally, we propose a novel metric for identifying motifs involved in epistasis and confirm our findings across three experimental datasets. Our work provides practical guidance for modeling and interpreting massively parallel reporter assay experiments from end to end.