Search bioRxiv⌕ Search

Biology subjects

Shahgir, H. S.

Publications and source records attributed to Shahgir, H. S..

2 recordsLinked to original sources

RNA-DCGen: Dual Constrained RNA Sequence Generation with LLM-Attack

Designing RNA sequences with specific properties is critical for developing personalized medications and therapeutics. While recent diffusion and flow-matching-based generative models have made strides in conditional sequence design, they face two key limitations: specialization for fixed constraint types, such as tertiary structures, and lack of flexibility in imposing additional conditions beyond the primary property of interest. To address these challenges, we introduce RNA-DCGen, a generalized framework for RNA sequence generation that is adaptable to any structural or functional properties through straightforward finetuning with an RNA language model (RNA-LM). Additionally, RNA-DCGen can enforce conditions on the generated sequences by fixing specific conserved regions. On RNA generation conditioned on RNA distance maps, RNA-DCGen generates sequences with an average R2 score of 0.625 compared to random sequences that score only 0.118 over 250 generations as judged by a separate more capable RNA-LM. When conditioned on RNA secondary structures, RNA-DCGen achieves an average F1 score of 0.4 against a random baseline of 0.006.

bioinformatics↗

BiRNA-BERT Allows Efficient RNA Language Modeling with Adaptive Tokenization

Recent advancements in Transformer-based models have spurred interest in their use for biological sequence analysis. However, adapting models like BERT is challenging due to sequence length, often requiring truncation for proteomics and genomics tasks. Additionally, advanced tokenization and relative positional encoding techniques for long contexts in NLP are often not directly transferable to DNA/RNA sequences, which require nucleotide or character-level encodings for tasks such as 3D torsion angle prediction. To tackle these challenges, we propose an adaptive dual tokenization scheme for bioinformatics that utilizes both nucleotide-level (NUC) and efficient BPE tokenizations. Building on the dual tokenization, we introduce BiRNA-BERT, a 117M parameter Transformer encoder pretrained with our proposed tokenization on 28 billion nucleotides across 36 million coding and non-coding RNA sequences. The learned representation by BiRNA-BERT generalizes across a range of applications and achieves state-of-the-art results in long-sequence downstream tasks and achieves a performance comparable to 6x larger models in short-sequence tasks with 27xless pre-training compute. BiRNA-BERT can dynamically adjust its tokenization strategy based on sequence lengths, utilizing NUC for shorter sequences and switching to BPE for longer ones, thereby offering, for the first time, the capability to efficiently handle arbitrarily long DNA/RNA sequences. 1

bioinformatics↗