UPLiFT: Uncertainty Parameterized Activation for Lightweight Fine-Tuning

Abstract

Learned protein representations contribute to the broad utility of pretrained protein language models (PLMs), often enhanced by task-specific fine-tuning where parameter-efficient techniques are imperative due to large network size. Our work takes an orthogonal approach to such fine-tuning: rather than optimizing the network weights, we formulate adaptation as an optimization task over the parameterized activation space. Motivated by the prior work on curvature-tuning, we propose UPLiFT, Uncertainty Parameterized activation for Lightweight Fine-Tuning, with enhanced adaptation capacity when the underlying model--whether pretrained or randomly initialized--remains fixed, for significantly improved parameter efficiency. Furthermore, its learnable non-linearity naturally allows to model uncertainty over the activation space through fine-tuning in a variational inference framework. Evaluation across multiple protein datasets shows that UPLiFT performs competitively while requiring 3-11x fewer trainable parameters than weight-only adaptations. Bayesian UPLiFT further improves calibration while largely preserving the predictive performance of deterministically fine-tuned models. The narrow predictive performance margin with the low-rank adaptation baseline in residue-level tasks highlights the potential of our proposed approach as a parameter-efficient fine-tuning technique for PLM-based predictors.