Search bioRxiv⌕ Search
Home/Archive/bioRxiv

bioRxiv · 10.1101/2024.12.10.627863

Strategies for robust, accurate, and generalizable benchmarking of drug discovery platforms

Van Norden, M.·Mangione, W.·Falls, Z.·Samudrala, R.
View original source ↗Export metadata ↓

Abstract

Benchmarking is essential for the improvement and comparison of drug discovery platforms. We revised the protocols used to benchmark our Computational Analysis of Novel Drug Opportunities (CANDO) multiscale therapeutic discovery platform to bring them into strong alignment with best practices.CANDO ranked 7.4% and 12.1% of known drugs in the top 10 compounds for their respective diseases/indications using drug-indication mappings from the Comparative Toxicogenomics Database (CTD) and Therapeutic Targets Database (TTD), respectively. Better performance was weakly correlated (Spearman correlation coefficient >0.3) with the number of drugs associated with an indication and moderately correlated (coefficient >0.5) with intra-indication chemical similarity. There was also a moderate correlation between performance on our original and new benchmarking protocols. Higher performance was observed when using TTD instead of CTD when drug-indication associations appearing in both mappings were assessed. CANDO is available at https://github.com/ram-compbio/CANDO. The version used in this paper is available at http://compbio.buffalo.edu/data/mc_cando_benchmarking2. Supplementary data, drug-indication interaction matrices, and drug-indication mappings are available at http://compbio.buffalo.edu/data/mc_cando_benchmarking2.

Source connections

Explore related subjects

bioinformaticsPreprint

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Van Norden, M., Mangione, W., Falls, Z., Samudrala, R.. 2024-12-16. Strategies for robust, accurate, and generalizable benchmarking of drug discovery platforms. https://doi.org/10.1101/2024.12.10.627863

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related preprints

bioRxiv2026

BiomiX 3.0: A user-friendly platform for democratized multi-omics integration with graph-based learning.

Background Multi-omics integration has emerged as a powerful strategy to decode the molecular complexity of biological systems. However, the diversity of available methods, each designed with distinct assumptions, objectives, and computational requirements, makes method selection, usage and interpretation challenging for nonexpert users. Here we present BiomiX 3.0, an updated version of the BiomiX platform that extends its integration capabilities with four additional methods: Similarity Network Fusion (SNF), NEighborhood-based Multi-Omics clustering (NEMO), Data Integration Analysis for Biomarker discovery using Latent variable approaches for Omics studies (DIABLO), and PRAMIGO (Phenotyping netwoRk Application for Multi-omics InteGratiOn), a novel supervised heterogeneous graph transformer (HGT) introduced in this work. Results We benchmarked all five methods, MOFA, DIABLO, SNF, NEMO, and PRAMIGO, on two independent multiomics datasets derived from a Chronic Lymphocytic Leukemia (CLL) cohort comparing IGHV-mutated and unmutated patients, and a pulmonary tuberculosis (PTB) cohort versus healthy controls. Supervised methods (DIABLO, PRAMIGO) consistently achieved higher condition-specific discrimination as measured by the Adjusted Rand Index (ARI) and the Adjusted Mutual Information (AMI). In contrast, unsupervised methods (SNF, NEMO) revealed alternative patient stratifications driven by independent sources of biological variance while MOFA performed in a semi-supervised way occupies an intermediate position, capturing latent factors that explain both disease-associated and orthogonal sources of variance. Systematic gene-centric analysis of the top-ranked features prioritized by each method was supported by manual biological annotation of shared and method-specific signals. Across cohorts, we annotated 154 shared features (116 genes, 38 metabolites) and 60 method-unique features per cohort, demonstrating that no single integration strategy captures the full landscape of biologically relevant signals. In the CLL cohort, shared features spanned B-cell receptor biology, innate immune signaling, RAS/MAPK activation, and epigenetic regulation, while methodunique features revealed supervised-method-specific insights into vesicle trafficking (DIABLO), immune checkpoints (MOFA), and ncRNA regulation (PRAMIGO). In the PTB cohort, a convergent interferon/innate immune signature dominated shared features across all methods. Still, method-unique analysis uncovered DIABLO-specific acylcarnitine metabolic reprogramming, MOFA-specific restoration of lysosomal trafficking, and PRAMIGO-specific {gamma}{delta} T-cell and immunoglobulin repertoire diversity. Across both cohorts, SNF and NEMO proved useful for detecting biological and technical sources of variation that were orthogonal to the primary condition of interest. Ultimately, PRAMIGO uniquely enables the construction of heterogeneous graphs modeling cross-modal molecular interactions, uncovering epigenetic co-regulation programs in CLL and multi-omics inflammatory modules in PTB that are difficult to identify using conventional integration approaches. Conclusions BiomiX 3.0 provides a graphical user interface (GUI) multi-method integration environment that democratizes access to state-of-the-art multi-omics analysis. By combining both unsupervised and supervised integration strategies within a unified platform and introducing graph-based learning through PRAMIGO, BiomiX 3.0 enables researchers across disciplines with complementary tools to interrogate the biological sources of variation in their data, without requiring bioinformatics expertise.

bioinformatics↗
bioRxiv2026

Perfect 21-nucleotide matches to beneficial fungi are common in canonical antifungal dsRNA targets: an in-silico off-target hazard screen for spray-induced gene silencing

Double-stranded RNA (dsRNA) biopesticides that silence essential fungal genes by spray-induced gene silencing (SIGS) are advancing towards registration, underpinned by the claim that silencing is confined to sequence-matched target organisms, a claim that has never been tested systematically against beneficial fungi. We screened twelve dsRNA constructs (the whole transcripts of eleven canonical antifungal target genes of Fusarium graminearum: three CYP51 sterol 14-demethylases, six chitin synthases and two {beta}-tubulins, plus a concatenation of the three CYP51 transcripts modelling the flagship CYP3RNA design) for perfect 21-nucleotide (21-nt) identity, the primary match criterion of recent regulatory bioinformatics frameworks, against the transcriptomes of seven non-target fungi, including commercial biocontrol agents (Trichoderma harzianum, T. virens, Beauveria bassiana, Metarhizium anisopliae, M. brunneum), the arbuscular mycorrhizal fungus Rhizophagus irregularis and Saccharomyces cerevisiae. Eleven of the twelve constructs carried off-target hazard; only CYP51C was completely clean. TUB2b carried 580 perfect 21-nt matches and contained no hazard-free window of 250 consecutive 21-mer start positions, whereas 26.6-60.0% of screened windows were hazard-free in the designable genes. Hits landed overwhelmingly on orthologues of the targeted gene and concentrated in the Hypocreales; R. irregularis and S. cerevisiae were nearly clean. These are hazard findings, not risk findings: perfect siRNA-length matches to beneficial fungi are common in canonical antifungal dsRNA targets, and current regulatory screening panels contain no fungus that would detect them.

bioinformatics↗
bioRxiv2026

Fitting dynamics is not identifying causal edges: a white-box masked ODE benchmark for trans-omics digital twins of drug action mechanisms

Multi-component drug regimens, with traditional Chinese medicine formulas as the hardest case, act across signalling, transcriptional, proteomic and metabolic layers, and elucidating their mechanisms requires dynamic models that predict molecular trajectories rather than static association networks. Trainable ordinary differential equation (ODE) systems fitted to time-series omics are increasingly used for this purpose; yet their validation remains fit-based: at genomic scale, no ground truth has existed to test whether a good fit implies correct mechanisms. Here we build that ground truth: a white-box masked ODE benchmark on the real trans-omics topology of insulin action in mouse liver (transcriptome GEO GSE166336, proteome ProteomeXchange PXD022728, phosphoproteome PXD022823, metabolome source-publication Tables S2-S3; 2,106 molecular species; 4,912 ground-truth edges), with controllable noise, missingness and sampling budgets. Four instruments quantify identifiability: an oracle-perturbation basin curve, a held-out-layer corruption assay, a saturation audit, and an ideal-budget ceiling test. We find that a static baseline (FD + LASSO) performs at chance (AUROC ~ 0.50, except GE at 0.567); that from-scratch training remains at chance even with noise-free, fully observed, densely sampled data (per-layer AUROC 0.48-0.53); that held out layers act as corruption sinks whose failure decomposes into an information floor, a scale-mismatch amplifier, and an edge-gradient drag, curable only jointly; that tanh saturation silently zeroes entire regulator columns; and that a 12-knockout validation battery decomposes intervention reliability by network distance. We distill these into operational prescriptions. The binding constraint is not fitting but structural identifiability. Benchmark, code and audit tools are planned for open release upon publication.

bioinformatics↗