Coarse-tuning to improve a protein foundation model for a domain of proteins: the example of outer membrane beta barrels

Abstract

Foundation models for proteins have shown great potential in addressing a wide range of challenges in computational molecular biology. State-of-the-art models are trained on large and diverse corpora of protein sequences, enabled by self-supervised training. The models are then used in transfer learning and the embeddings they calculate are used in supervised downstream tasks. We propose a novel focusing mechanism for foundation models, which we call 'coarse-tuning'. In coarse-tuning, we further update the model, using a defined yet unlabeled, dataset to enhance its performance in a specific domain. We demonstrate this approach using Outer Membrane Beta Barrels (OMBBs) - proteins that are found in the distinct biophysical environment of the outer membrane. Our results show that coarse-tuned models outperform the original foundation model when applied to various transfer learning tasks. Additionally, we study the embeddings generated by the coarse-tuned models and compare them to those from the original foundation model, including an investigation of the learned attention matrices via contact map predictions. Our work suggests that a medium-sized dataset (fewer than 100,000 protein sequences) is sufficient to effectively specialize a foundation model to a specific domain, enhancing its utility for domain-specific applications.