MuLan-Methyl - Multiple Transformer-based Language Models for Accurate DNA Methylation Prediction

Abstract

Transformer-based language models are successfully used to address massive text-related tasks. DNA methylation is an important epigenetic mechanism and its analysis provides valuable insights into gene regulation and biomarker identification. Several deep learning-based methods have been proposed to identify DNA methylation and each seeks to strike a balance between computational effort and accuracy. Here, we introduce MuLan-Methyl, a deep-learning framework for predicting DNA methylation sites, which is based on five popular transformer-based language models. The framework identifies methylation sites for three different types of DNA methylation, namely N6-adenine, N4-cytosine, and 5-hydroxymethylcytosine. Each of the employed language models is adapted to the task using the "pre-train and fine-tune" paradigm. Pre-training is performed on a custom corpus of DNA fragments and taxonomy lineages using self-supervised learning. Fine-tuning aims at predicting the DNA-methylation status of each type. The five models are used to collectively predict the DNA methylation status. We report excellent performance of MuLan-Methyl on a benchmark dataset. Moreover, we argue that the model captures characteristic differences between different species that are relevant for methylation. This work demonstrates that language models can be successfully adapted to applications in biological sequence analysis and that joint utilization of different language models improves model performance. Mulan-Methyl is open source and we provide a web server that implements the approach. Key pointsO_LIMuLan-Methyl aims at identifying three types of DNA-methylation sites. C_LIO_LIIt uses an ensemble of five transformer-based language models, which were pre-trained and fine-tuned on a custom corpus. C_LIO_LIThe self-attention mechanism of transformers give rise to importance scores, which can be used to extract motifs. C_LIO_LIThe method performs favorably in comparison to existing methods. C_LIO_LIThe implementation can be applied to chromosomal sequences to predict methylation sites. C_LI