scDynOmics: An Optimized Transformer Model for Representation Learning from Single-Cell Multiomics

Abstract

As foundation models have become increasingly prevalent in several fields for multiple purposes, pretraining models with single-cell transcriptomic data has gained significant interest. Although existing single-cell foundation models have demonstrated that transformer-based designs can be applied to various biological tasks, they do not show consistently competitive performance compared to much simpler approaches while requiring much more resources in terms of data and compute. Here, we introduce scDynOmics, a pretrainable multiomics-capable transformer for representation learning from single-cell data. The model is motivated by gene regulatory networks without excluding unknown interactions between genes and adopts a Linformer-style attention mechanism to scale to coding-genome wide multimodal inputs. Pretraining on paired single-cell transcriptomic and chromatin accessibility profiles yields compact high-fidelity embeddings that represent cellular states and developmental dynamics. For versatile application, scDynOmics employs low-rank adaptation modules, enabling parameter-efficient fine-tuning for downstream tasks. We demonstrate that scDynOmics outperforms existing single-cell foundation models by a large margin and achieves or surpasses state-of-the-art performance compared to simpler approaches, while revealing interpretable factors driving developmental trajectories and perturbation responses that simpler approaches cannot provide. Overall, scDynOmics is an efficient, scalable, flexible, and interpretable framework for cellular representation learning and deciphering cellular heterogeneity and dynamics.