Spontaneous Emergence of Symmetry in a Generative Model of Protein Structure
Generative models are becoming powerful tools for protein design, enabling the creation of novel protein structures and sequences. Recent approaches have shown success using diffusion models and flow matching to sample realistic protein folds. Many methods explicitly incorporate structural biases or constraints--for example, enforcing symmetry during generation--to steer the design process. Here we report the spontaneous emergence of structural symmetry in a transformer-based generative model for proteins, without any symmetry-specific conditioning during training or constraints during generation. In our flow-matching model, a single attention head in an SE(3)-equivariant transformer layer was found to be primarily responsible for the models ability to generate symmetric arrangements of backbone residues across chains, or repeating motifs within a chain. Our results show that protein generative models can learn high-level structural patterns implicitly from training data. This opens new questions about interpretability and control in generative design: understanding how and why a single attention head can govern a complex global property like symmetry may inform future model architectures and help exploit emergent behaviors for better protein engineering.