Uncovering Features of ProteinMPNN with Sparse Autoencoders

Abstract

Customization and modification of machine learning derived sequences can be achieved by understanding the mechanics behind state-of-the-art protein design models such as ProteinMPNN. Uninterpretability is hypothesized to be due largely to polysemanticity where neural networks store more features than they have dimensions. Here, we introduce sparse autoencoders (SAEs) into ProteinMPNN which expand its latent space and uncovers correlations between the activations in sparse dimensions and the presence of protein features such as secondary structure, residue volume, residue shape, disulfide bridges, and coordination to ligands. Additionally, we show that sparse activations can be manipulated to produce a variant of the short chain variable fragment Brolucizumab that is more thermostable than variants produced by traditional machine learning methods based on high temperature molecular dynamics simulations. Thus, SAEs provide both mechanistic understanding of protein design models and allow customizable manipulation of sequence prediction.