Estimating the replicability of Brazilian biomedical science

Abstract

Concerns over the replicability and reproducibility of published research have grown in many research fields, but empirical data to inform policies are still scarce. Biomedical research in Brazil expanded rapidly over the last three decades, with no systematic assessment of the replicability of its findings. With this in mind, we set up the Brazilian Reproducibility Initiative, a multicenter replication of published experiments from Brazilian science using three common experimental methods: the MTT assay, the reverse transcription polymerase chain reaction (RT-PCR) and the elevated plus maze (EPM). A total of 56 laboratories performed 143 replications of 56 experiments; of these, 90 replications of 45 experiments were considered valid by an independent committee. Replication rates for these experiments varied between 20 and 44% according to five predefined criteria. In median terms, ratios between group means were 58% lower in replications than in original experiments, while coefficients of variation were 82% higher. Effect size decrease was smaller for MTT experiments, original results with less variability and those considered more challenging to replicate, while t values for replications were positively correlated with researcher predictions about replicability, and negatively correlated with the rate of publications by the original articles last author. Deviations from preregistered protocols were very common in replications, most frequently due to reasons inherent to the experimental model or related to infrastructure and logistics. Our results highlight factors that limit the replicability of results published by researchers in Brazil and suggest ways by which this scenario can be improved.