Widespread use of invalid statistical tests in biomedical machine learning

Zeng, T.; Li, H.; Zhang, S.; Tan, Y. Q.; Tian, F.; Orban, C.; An, L.; Che, W.; Cheng, J.; Chong, J. S. X.; Dehestani, N.; Dong, Z.; Li, X.; Li, Z.; Lim, M. J. R.; Lin, Y.; Ling, Q.; Ling, Z.; Low, X. Z.; Mansour L., S.; Ng, K. K.; Nguyen, T. T.; Ooi, L. Q. R.; Pande, S.; Qian, X.; Ruan, J.; Wang, Z.; Xie, Y.; Zhang, C.; Zhang, Y.; Patil, K.; Parkes, L.; Dhamala, E.; Chopra, S.; Zalesky, A.; Holmes, A.; Eickhoff, S.; Zhou, J. H.; Renaud, O.; Dosenbach, N.; Kording, K. P.; Bzdok, D.; Nichols, T.; Yeo, B. T. T.

doi:10.64898/2026.05.17.724301

Home/Archive/bioRxiv

bioRxiv · 10.64898/2026.05.17.724301

Widespread use of invalid statistical tests in biomedical machine learning

View original source ↗Export metadata ↓

Abstract

Cross-validation is routinely used to compare performance in biomedical artificial intelligence. Standard tests ignore correlation across cross-validation folds, inflating false-positive rates. In a PRISMA-guided meta-analysis of 184 studies (impact factor [≥] 15) across 30 biomedical fields, 97% use invalid tests. Among studies with abstract-level claims supported by invalid tests, 59% rely on a spurious comparison: one that loses significance after correcting for this correlation. On average, regaining significance requires a 43% larger effect. Extending these findings to the broader literature, we estimate that spurious comparisons occur in nearly two in three studies and support abstract-level claims in nearly one in three studies. Simulations confirm false-positive rates paradoxically approach 100% when cross-validation is repeated to improve stability. We introduce SHARP, a redesign of cross-validation, which best balances false-positive control and power among 13 benchmarked tests. These results reveal widespread fragility in biomedical artificial intelligence and offer a practical route to valid comparisons.

Explore related subjects

bioinformatics Preprint

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Zeng, T., Li, H., Zhang, S., Tan, Y. Q., Tian, F., Orban, C., An, L., Che, W., Cheng, J., Chong, J. S. X., Dehestani, N., Dong, Z., Li, X., Li, Z., Lim, M. J. R., Lin, Y., Ling, Q., Ling, Z., Low, X. Z., Mansour L., S., Ng, K. K., Nguyen, T. T., Ooi, L. Q. R., Pande, S., Qian, X., Ruan, J., Wang, Z., Xie, Y., Zhang, C., Zhang, Y., Patil, K., Parkes, L., Dhamala, E., Chopra, S., Zalesky, A., Holmes, A., Eickhoff, S., Zhou, J. H., Renaud, O., Dosenbach, N., Kording, K. P., Bzdok, D., Nichols, T., Yeo, B. T. T.. 2026-05-20. Widespread use of invalid statistical tests in biomedical machine learning. https://doi.org/10.64898/2026.05.17.724301

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related preprints

bioRxiv2026

Accounting for pseudo-replication of Linkage Disequilibrium for contemporary Ne estimation

The Linkage Disequilibrium (LD) of unlinked loci can be used to estimate contemporary effective population size (Ne) of one to a few generations ago. In genomic datasets loci on different chromosomes are considered unlinked, but there are many more pairs of unlinked loci than there are independent pairs of chromosomes, resulting to confidence intervals (C.I.) being too narrow if the non-independence is not taken into account. Simulations were run to investigate the correlation structure among LD of unlinked loci, which can be expressed by the LD of loci along the same chromosomes, based on a discovery of a novel Random Probe LD estimator. We classify the correlation into two categories: overlapping of loci and disjoint pairs. The former is induced from the same locus being considered twice and is the stronger form of correlation. These correlations feed into {rho}, a parameter to quantify the degree of pseudo-replication in a dataset, and further a correction formula from which C.I. can be properly inferred. We demonstrate the use of our method via an analysis of genomic data from the malaria-transmitting Anopheles gambiae s.s mosquitoes. Apart from the point and C.I. estimates, we find that Var((r^2 ) ) is inflated by about 550 times due to pseudo-replication, highlighting the danger of not handling genetic correlation properly.

bioinformatics↗

bioRxiv2026

Accurate and scalable decontamination of imaging-based spatial transcriptomics via optimal transport

Imaging-based spatial transcriptomics enables molecule-resolved profiling of gene expression and tissue organization in situ. However, segmentation errors, transcript spillover and three-dimensional cell overlap can introduce misassigned transcripts into cell-level expression profiles, compromising biological interpretation and obscuring genuine signals. Existing methods either remove suspect expression at the cost of signal loss or lack a biologically grounded criterion for transcript assignment. Here we present CellDot, an optimal-transport framework that determines the fate of each transcript by retaining it in its host cell, reassigning it to a plausible neighboring cell or removing it as background. By integrating reference-guided expression compatibility with spatial information and data-adaptive constraints, CellDot enables accurate and traceable molecule-level correction while preserving biologically meaningful variation. In evaluations across multiple human tumor datasets, CellDot exhibited superior performance compared to existing decontamination methods, successfully restoring spatial expression patterns that matched independent cross-platform measurements. Moreover, it significantly enhanced the recovery of cellular states, intercellular communication, and spatial niche programs. Our experiments using real data demonstrated CellDot's scalability and established it as the only method applicable to a whole-transcriptome Atera dataset, underscoring its distinct advantages in the field of spatial transcriptomics.

bioinformatics↗

bioRxiv2026

Interpretable Machine Learning Reveals Complementary Age-Related Signatures in the Oral and Gut Microbiome

Whether combining microbiome data from multiple body sites improves prediction, and whether different sites carry complementary or redundant information, are distinct questions that most studies conflate into a single accuracy metric. This work makes two contributions, one methodological and one biological, using paired stool and oral cavity microbiome samples from 44 subjects across two age groups, healthy adults and newborns (Ferretti et al., 2018). Methodologically, we show that a subject-matched fusion design combined with SHAP-based (SHapley Additive exPlanations) site attribution can detect complementary information between body sites even when no measurable accuracy gain results. This is a pattern that conventional model comparison would misread as a null result. Gut (stool) composition alone achieved near-perfect classification (area under the receiver operating characteristic curve, AUC = 1.00), and combined stool-oral models never exceeded this ceiling. A null baseline, bootstrap confidence intervals, and preprocessing sensitivity checks confirmed that this ceiling reflects genuine biological signal rather than an artifact. Despite the flat accuracy curve, SHAP analysis of the fused model showed that oral cavity features carried more total feature importance than stool features (58.1% versus 41.9%), indicating that the model draws on real, non-redundant information from both sites. Biologically, the taxa driving this pattern include Malassezia restricta, Staphylococcus epidermidis, and Prevotella melaninogenica. These taxa behave in a manner consistent with their established roles as early colonizers of the neonatal gut, skin, and oral cavity, once their model-specific behavior is verified directly against abundance data rather than inferred from the literature alone. An independent, substantially larger paired-cohort study using a different analytical method reports a compatible pattern. Together, these results support a model of oral-gut microbiome maturation as two distinct, complementary processes, and demonstrate that detecting this kind of relationship requires examining a model's internal reasoning rather than its accuracy alone.

bioinformatics↗