Biology subjects

O'Connell, K. A.

Publications and source records attributed to O'Connell, K. A..

4 recordsLinked to original sources

bioRxiv2026

ClaroAI-Bench: Evaluating Agentic Scientific Reproducibility on Real Biomedical Papers

We introduce ClaroAI-Bench, an evaluation suite for measuring AI agents ability to reproduce computational findings from published biomedical research. The benchmark comprises 35 real NIH-funded papers spanning five modalities (genomics, imaging, clinical/EHR, epidemiology, wet-lab) scored on a five-dimension rubric: data findability (D1), data accessibility (D2), code availability (D3), environment reconstructability (D4), and results reproducibility (D5). Each task requires an agent to locate data, obtain code, reconstruct the compute environment, execute the analysis, and verify results against published claims--mirroring the full scientific reproduction pipeline. In a three-condition ablation, an audit-only baseline (D1-D4 metadata scoring) and a bash-only agent (API + bash tool) both achieve 0% D5 reproduction, while a full-capability agent (Claude Code, all tools) reproduces 20 of 33 computational papers (60.6%; 95% CI [42.4, 75.8]). D1-D4 metadata scores strongly predict D5 outcomes (Spearman r=0.68, p<0.0001), and papers with accessible data and code achieve 2.9x higher D5 scores than restricted papers (p=0.0013). Multi-model scoring with three frontier models (Claude Opus 4.6, GPT-5.4, Gemini 2.5 Pro) yields inter-model agreement of r=0.85-0.97 on D3 but only r=0.51-0.81 on D4, identifying environment reconstruction as the dimension with highest evaluator disagreement. ClaroAI-Bench fills a gap between code-generation benchmarks (SWE-bench) and end-to-end scientific AI evaluations by testing long-horizon, real-world reproduction tasks with brittle environments, missing meta-data, and access constraints. The benchmark, scoring rubric, agent logs, and pip-installable auditor are archived at https://doi.org/10.5281/zenodo.20071236 and on HuggingFace Datasets at https://huggingface.co/datasets/kyleaoconnell22/claroai-bench.

bioinformatics↗

bioRxiv2026

Signal Versus Noise: Evaluating iNaturalist Photos as a Source of Quantitative Phenotypic Data in Plethodon Salamanders using Autoresearch and Agentic AI

Community-science platforms such as iNaturalist now contain tens of millions of georeferenced, photographically vouchered biodiversity records, yet extracting reliable quantitative measurements from opportunistic photographs remains methodologically challenging. Here, I evaluate the signal-to-noise ratio of iNaturalist photos for phenotyping Plethodon salamanders across two trait classes: continuous dorsal brightness (a proxy for ecogeographic clines predicted by Glogers rule and the thermal melanism hypothesis) and discrete color morph frequency in P. cinereus. I optimized a color-extraction pipeline using an agent-guided parameter search adapted from the autoresearch framework (Karpathy 2026; Schmidgall et al. 2025), exploring crop fraction, color space, normalization, and quality-control thresholds across 50 bounded micro-experiments. Applying the production HSV pipeline to 103,653 observations of 34 species, I found negligible geographic structure in dorsal brightness (R2 = 0.001), even within P. cinereus alone (n = 71,627). Variance decomposition showed that photographer identity explains 23.3% of brightness variance, geography 5.1%, species 1.6%, and time of day 0.3%, with 69.7% residual. In contrast, a hue-threshold morph classifier recovered a significant geographic signal in red-back frequency (R2 = 0.008, p < 0.001), 7x stronger than the brightness result, though still weaker than the supervised CNN of Hantak et al. (2022; pseudo-R2 {approx} 0.04). These results indicate that citizen-science photographs are poorly suited to continuous quantitative phenotyping under current collection conditions, whereas discrete categorical traits remain recoverable with appropriate classifiers. The autoresearch loop clarified the failure mode: no tested parameter configuration recovered a meaningful brightness signal from a dataset dominated by observer effects.

ecology↗

bioRxiv2022

Accelerating genomic workflows using NVIDIA Parabricks

BackgroundAs genome sequencing becomes a more integral part of scientific research, government policy, and personalized medicine, the primary challenge for researchers is shifting from generating raw data to analyzing these vast datasets. Although much work has been done to reduce compute times using various configurations of traditional CPU computing infrastructures, Graphics Processing Units (GPUs) offer the opportunity to accelerate genomic workflows by several orders of magnitude. Here we benchmark one GPU-accelerated software suite called NVIDIA Parabricks on Amazon Web Services (AWS), Google Cloud Platform (GCP), and an NVIDIA DGX cluster. We benchmarked six variant calling pipelines, including two germline callers (HaplotypeCaller and DeepVariant) and four somatic callers (Mutect2, Muse, LoFreq, SomaticSniper). ResultsFor germline callers, we achieved up to 65x acceleration, bringing HaplotypeCaller runtime down from 36 hours to 33 minutes on AWS, 35 minutes on GCP, and 24 minutes on the NVIDIA DGX. Somatic callers exhibited more variation between the number of GPUs and computing platforms. On cloud platforms, GPU-accelerated germline callers resulted in cost savings compared with CPU runs, whereas somatic callers were often more expensive than CPU runs because their GPU acceleration was not sufficient to overcome the increased GPU cost. ConclusionsGermline variant callers scaled with the number of GPUs across platforms, whereas somatic variant callers exhibited more variation in the number of GPUs with the fastest runtimes, suggesting that these workflows are less GPU optimized and require benchmarking on the platform of choice before being deployed at production scales. Our study demonstrates that GPUs can be used to greatly accelerate genomic workflows, thus bringing closer to grasp urgent societal advances in the areas of biosurveillance and personalized medicine.

bioinformatics↗

bioRxiv2019

Congruent demographic responses to Pleistocene geological processes in Sumatran parachuting frogs: a comparison of target-capture and ddRADseq for population genomic analyses

Catastrophic events, such as volcanic eruptions, can have profound impacts on the demographic histories of resident taxa. Due to its presumed effect on biodiversity, the Pleistocene eruption of super-volcano Toba has received abundant attention. We test the effects of the Toba eruption on the diversification, genetic diversity, and demography of three co-distributed species of parachuting frogs (Genus Rhacophorus) on Sumatra. We generate target-capture data ([~]950 loci and [~]440,000 bp) for three species of parachuting frogs and use these data paired with previously generated double digest restriction-site associated DNA (ddRADseq) data to estimate population structure and genetic diversity, to test for population size changes using demographic modelling, and to estimate the temporal clustering of size change events using a full-likelihood Bayesian method. We find that populations around Toba exhibit reduced genetic diversity compared with southern populations, and that northern populations exhibit a shift in effective population size around the time of the eruption ([~]80 kya). However, we infer a stronger signal of expansion in southern populations around [~]400 kya, and at least two of the northern populations may have also expanded at this time. Taken together, these findings suggest that the Toba eruption precipitated population declines in northern populations, but that the demographic history of these three species was also strongly impacted by mid-Pleistocene forest expansion during glacial periods. We propose local rather than regional effects of the Toba eruption, and emphasize the dynamic nature of diversification on the Sunda Shelf.

evolutionary biology↗