Benchmarking virus identification tools on vaginal metagenomes

Abstract

The vaginal microbiome is associated with reproductive and sexual health, yet its viral component remains understudied. A recent vaginal genome catalogue reported that 85.8% of its species-level viral groups were absent from all five viral databases examined. Viral identification tools depend on reference sequences, either directly or through their training data. Previous benchmarks used simulated or non-vaginal datasets. How well these tools generalise to vaginal metagenomes therefore remains unclear. We benchmarked 14 tools spanning five methodological approaches. Evaluations used controlled genome fragments, viral spike-in co-assemblies and virus-like particle (VLP)-enriched co-assemblies. A multi-evidence benchmark assessed 13 vaginal shotgun assemblies (primary cohort). An independent 30-sample cohort provided external validation. geNomad ranked first or second by Matthews correlation coefficient (MCC) in every ranked evaluation. VIBRANT achieved similar MCC to geNomad on the primary assemblies. It made eight false-positive calls among 11,577 negatives but recovered only 22% of viral contigs. VIBRANT ranked first in the independent cohort. Tool rankings agreed closely between cohorts (Spearman rho = 0.952; 95% CI 0.921 - 0.960). Detection of sequences lacking reference homologues differed substantially between tools. Two phage genomes without detectable reference homologues provided a controlled novelty test. Each evaluated composition-based tool recovered at least 74% of their 1,500-bp fragments. Marker- or reference-dependent tools recovered at most 31%. A separate analysis examined 48 unconfirmed candidate viral contigs lacking detectable nucleotide or protein homologues. Jaeger flagged 41, compared with four for VIBRANT and one for geNomad. On short fragments, attention-based tools achieved 4.0-5.5 times the mean area under the precision-recall curve of feature-based tools. ViraLM accounted for most of this difference. ViraLM also had the highest recall (sensitivity) of eukaryote-infecting viruses in the evaluated panel. VirSorter2 achieved the highest MCC on VLP-enriched contigs. These results support geNomad when prioritising overall classification performance and VIBRANT when minimising false-positive phage calls is the priority. ViraLM supports short-contig and eukaryote-infecting-virus detection. VirSorter2 supports identification in VLP-enriched assemblies, and Jaeger supports screening for candidate novel phages. These findings guide tool selection for studies investigating the vaginal virome in reproductive and sexual health.