That's not a Hybrid: How to Distinguish Patterns of Admixture and Isolation-by-Distance
Describing naturally occurring genetic variation is a fundamental goal of molecular phylogeography and population genetics. Popular methods for this task include STRUCTURE, a model-based algorithm that assigns individuals to genetic clusters, and Principal Component Analysis (PCA), a parameter-free method. The ability of STRUCTURE to infer mixed ancestry makes it popular for documenting natural hybridization, which is of considerable interest to evolutionary biologists, given that such systems provide a window into the speciation process. Yet STRUCTURE can produce misleading results when its underlying assumptions are violated, like when genetic variation is distributed continuously. To test the ability of STRUCTURE and PCA to accurately distinguish admixture from continuous variation, we use forward-time simulations to generate population genetic data under three demographic scenarios: two involving admixture and one with isolation-by-distance (IBD). STRUCTURE and PCA alone cannot distinguish admixture from IBD, but complementing these analyses with triangle plots, which visualize hybrid index against interclass heterozygosity, provides more accurate inference of demographic history. We demonstrate that triangle plots are robust to missing data, while STRUCTURE and PCA are not, and show that setting a low allele frequency difference threshold for AIM identification can accurately characterize the relationship between hybrid index and interclass heterozygosity across demographic histories of admixture and range expansion. While STRUCTURE and PCA provide useful summaries of genetic variation, results should be paired with triangle plots before admixture is inferred.