TCR2vec: a deep representation learning framework of T-cell receptor sequence and function

Abstract

The T-cell receptor (TCR) repertoires are critical components of the adaptive immune system, and machine learning methods were proposed to analyze the TCR repertoire data. However, most methods work solely on the hypervariable CDR3 regions of TCRs, overlooking the information encoded in other domains. Representing full TCRs as informative vectors can be indispensable for developing reliable and effective machine learning models. We introduce TCR2vec, a deep representation learning framework with 12 layers of transformer blocks, to pave the way for downstream modelings of full TCRs. Together with masked language modeling (MLM), we propose a novel pretraining task named similarity preservation modeling (SPM) to capture the sequence similarities of TCRs. Through a multi-task pretraining procedure on MLM and SPM, TCR2vec learns a contextual understanding of TCRs within a similarity-preserved vector space. We first verify the effectiveness of TCR2vec in predicting TCRs binding specificity and TCR clustering through comparison with three other embedding approaches. TCR2vec can be finetuned on small task-specific labeled data for enhanced performance, which outperforms state-of-the-art models by 2-25% in predicting TCRs binding specificity. Next, we compare the performance of two versions of TCR2vec pretrained on full TCRs (TCR2vec) or CDR3s (CDR3vec) and demonstrate that TCR2vec consistently outperforms CDR3vec by 12-14%. Further analysis of attention maps reveals that residues outside CDR3 also make notable contributions to the recognition of antigens. TCR2vec is available at https://github.com/jiangdada1221/TCR2vec.