HEIMDALL: A Modular Framework for Tokenization in Single-Cell Foundation Models

Abstract

Foundation models for single-cell RNA-sequencing (scRNA-seq) data are emerging as powerful tools for single-cell analysis, yet their performance depends critically on how cells are tokenized into model inputs. Single-cell data lack a canonical tokenization scheme, and many design choices in current single-cell foundation models (scFMs) remain heuristic, entangled, and difficult to evaluate. Here, we introduce HO_SCPLOWEIMDALLC_SCPLOW, a unified framework for dissecting and redesigning tokenizers in scFMs. By decomposing existing tokenization strategies into individual design choices, HO_SCPLOWEIMDALLC_SCPLOW enables attribution of the components that underlie robust generalization, allowing more principled design of improved tokenizers. Combining HO_SCPLOWEIMDALLC_SCPLOW with a minimal transformer backbone, we find that tokenizer design is instrumental for generalization in challenging distribution-shift settings such as cross-tissue, cross-species, and cross-gene-panel cell type classification, as well as reverse perturbation prediction. We show that, while tokenizer choice has little effect in scenarios with matched train and test data, it becomes imperative under distribution shift. Rather than identifying a single globally optimal tokenizer, HO_SCPLOWEIMDALLC_SCPLOW reveals that robust transfer depends on a small number of tokenization design axes - especially gene identity, expression encoding, and ordering - that expose different biological priors to the model. In this sense, universal transferability in scFMs still depends on a non-universal tokenizer interface. Together, these findings establish tokenization as a critical design axis in scFMs and provide design principles and reusable infrastructure for more robust scFMs.