BDQC: a general-purpose analytics tool for domain-blind validation of Big Data

Deutsch, E.; Kramer, R.; Ames, J.; Bauman, A.; Campbell, D. S.; Chard, K.; Clark, K.; D'Arcy, M.; Dinov, I.; Donovan, R.; Foster, I.; Heavner, B. D.; Hood, L. E.; Kesselman, C.; Madduri, R.; Mi, H.; Muruganujan, A.; Pa, J.; Price, N. D.; Robinson, M.; Sepehrband, F.; Toga, A. W.; Van Horn, J. D.; Zhao, L.; Glusman, G.

doi:10.1101/258822

Home/Archive/bioRxiv

bioRxiv · 10.1101/258822

BDQC: a general-purpose analytics tool for domain-blind validation of Big Data

View original source ↗Export metadata ↓

Abstract

Translational biomedical research is generating exponentially more data: thousands of whole-genome sequences (WGS) are now available; brain data are doubling every two years. Analyses of Big Data, including imaging, genomic, phenotypic, and clinical data, present qualitatively new challenges as well as opportunities. Among the challenges is a proliferation in ways analyses can fail, due largely to the increasing length and complexity of processing pipelines. Anomalies in input data, runtime resource exhaustion or node failure in a distributed computation can all cause pipeline hiccups that are not necessarily obvious in the output. Flaws that can taint results may persist undetected in complex pipelines, a danger amplified by the fact that research is often concurrent with the development of the software on which it depends. On the positive side, the huge sample sizes increase statistical power, which in turn can shed new insight and motivate innovative analytic approaches. We have developed a framework for Big Data Quality Control (BDQC) including an extensible set of heuristic and statistical analyses that identify deviations in data without regard to its meaning (domain-blind analyses). BDQC takes advantage of large sample sizes to classify the samples, estimate distributions and identify outliers. Such outliers may be symptoms of technology failure (e.g., truncated output of one step of a pipeline for a single genome) or may reveal unsuspected \" signal\" in the data (e.g., evidence of aneuploidy in a genome). We have applied the framework to validate real-world WGS analysis pipelines. BDQC successfully identified data outliers representing various failure classes, including genome analyses missing a whole chromosome or part thereof, hidden among thousands of intermediary output files. These failures could then be resolved by reanalyzing the affected samples. BDQC both identified hidden flaws as well as yielded new insights into the data. BDQC is designed to complement quality software development practices. There are multiple benefits from the application of BDQC at all pipeline stages. By verifying input correctness, it can help avoid expensive computations on flawed data. Analysis of intermediary and final results facilitates recovery from aberrant termination of processes. All these computationally inexpensive verifications reduce cryptic analytical artifacts that could seriously preclude clinical-grade genome interpretation. BDQC is available at https://github.com/ini-bdds/bdqc.

Explore related subjects

bioinformatics Preprint

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Deutsch, E., Kramer, R., Ames, J., Bauman, A., Campbell, D. S., Chard, K., Clark, K., D'Arcy, M., Dinov, I., Donovan, R., Foster, I., Heavner, B. D., Hood, L. E., Kesselman, C., Madduri, R., Mi, H., Muruganujan, A., Pa, J., Price, N. D., Robinson, M., Sepehrband, F., Toga, A. W., Van Horn, J. D., Zhao, L., Glusman, G.. 2018-02-02. BDQC: a general-purpose analytics tool for domain-blind validation of Big Data. https://doi.org/10.1101/258822

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related preprints

bioRxiv2026

Cross-modal transfer enables zero-shot prediction of drug responses across cell types

Genetic perturbation responses capture the functional consequences of gene-level interventions, providing functional information for drug-response prediction beyond baseline gene expression. Within the same cellular context, genetic and chemical perturbations engage shared regulatory networks, providing a biological basis for transferring response information between perturbation modalities. Here we introduce GRAFT, a framework that combines genetic-to-chemical transfer with cross-context prediction to infer drug-induced transcriptional responses without drug measurements in the target context. Across benchmarked LINCS and external contexts, GRAFT improved response correlation, directional agreement and magnitude reconstruction over competing methods, particularly for drugs with context-dependent responses. GRAFT-Adapt further refines predictions of unmeasured responses using a small number of drug measurements from the target context. GRAFT-Atlas spans 13 cellular contexts and 7,856 drug catalog entries, comprising 102,128 predictions for drug-context combinations. The atlas captured recognizable pharmacological neighborhoods, shared functional programs, context-dependent transcriptional changes and structural-functional associations supported by molecular docking. Across ten iPSC lines from distinct donors, GRAFT identified candidate inter-donor differences, with functional support for selected candidates from measured genetic perturbation responses. Together, GRAFT enables perturbation-informed prediction and comparison of drug responses across cellular contexts.

bioinformatics↗

bioRxiv2026

Interpreting graph neural networks reveals microbial association patterns in inflammatory bowel diseases

Traditional machine learning models trained on gut microbiome abundance profiles can distinguish individuals with inflammatory bowel disease (IBD) from non-IBD individuals. However, their interpretation typically considers taxa in isolation, making it difficult to understand how associations between taxa contribute to predictions. Here, we inferred individual-specific microbial association networks from longitudinal metagenomics data of 106 individuals in HMP2/iHMP and used them to train and interpret graph neural network (GNN) models, which we compared with abundance-based models such as Random Forest (RF). Although RF achieved higher predictive performance (AUPRC ~0.59 vs 0.446), interpreting the GNNs revealed that their predictions relied on a densely connected module of co-abundant Bacteroidales species. Unlike the taxa prioritized by RF, these species were not differentially abundant between conditions. In an independent cohort, this module carried most of the predictive signal, and removing its associations reduced performance to near chance. These results show that coordinated association structure, rather than abundance shifts alone, is a feature of the IBD gut microbiome, and that interpreting GNNs can uncover such patterns that complement abundance-based analyses.

bioinformatics↗

bioRxiv2026

FateDriver: Weakly supervised dual-view graph learning for transition-specific prioritization of candidate regulators in cellular reprogramming

Cellular reprogramming supports regenerative medicine, yet the endogenous regulators of a specified source-to-target transition remain incompletely defined. Existing computational methods rank candidate regulators from network topology, trajectory dynamics, or cell-state labels, leaving unused the factors experimentally delivered to induce the transition. We present FateDriver, a weakly supervised dual-view graph neural network that jointly represents cells and genes from single-cell transcriptomes, treats the delivered factors as sparse gene-level labels, and ranks the remaining genes for a specified transition under a positive--unlabeled formulation. Across two OSKM datasets and one GMT dataset, FateDriver ranked first or second among six methods in recovering externally curated reference genes, exceeded a supervision-matched non-graph model despite weaker co-expression with the delivered factors, and achieved the highest recovery of protein targets of pluripotency-inducing chemical cocktails in both OSKM datasets. Holding either the source or the endpoint fixed yielded distinct candidate programs, including a DDX58--interferon axis associated with cells lacking endogenous pluripotency markers. Applying the same framework to candidate source--target pairs prioritized distinct candidate origins for two in vivo partially reprogrammed intestinal populations. FateDriver converts sparse experimental knowledge into transition-specific hypotheses for regulator prioritization from single-cell trajectories.

bioinformatics↗