Judgments of other bias in Cochrane systematic reviews of interventions are highly inconsistent and thus hindering use and comparability of evidence

Babic, A.; Pijuk, A.; Brazdilova, L.; Georgieva, Y.; Raposo Pereira, M. A.; Poklepovic, T.; Puljak, L.

doi:10.1101/366591

Home/Archive/bioRxiv

bioRxiv · 10.1101/366591

Judgments of other bias in Cochrane systematic reviews of interventions are highly inconsistent and thus hindering use and comparability of evidence

Babic, A.·Pijuk, A.·Brazdilova, L.·Georgieva, Y.·Raposo Pereira, M. A.·Poklepovic, T.·Puljak, L.

View original source ↗Export metadata ↓

Abstract

BackgroundClinical decisions are made based on Cochrane systematic reviews (CSRs), but implementation of results of evidence syntheses such as CSRs is problematic if the evidence is not prepared consistently. All systematic reviews should assess risk of bias (RoB) in included studies, and in CSRs this is done by using Cochrane RoB tool. However, the tool is not necessarily applied according to the instructions. In this study we aimed to analyze types and judgments of other bias in the RoB tool in CSRs of interventions.\n\nMethodsWe analyzed CSRs that included randomized controlled trials (RCTs) and extracted data regarding other bias from the RoB table and accompanying support for the judgment. We categorized different types of other bias.\n\nResultsWe analyzed 768 CSRs that included 11369 RCTs. There were 602 (78%) CSRs that had other bias domain in the RoB tool, and they included a total of 7811 RCTs. In the RoB table of 337 CSRs for at least one of the included trials it was indicated that no other bias was found and supporting explanations were inconsistently judged as low, unclear or high RoB. In the 524 CSRs that described various sources of other bias there were 5762 individual types of explanations which we categorized into 31 groups. The judgments of the same supporting explanations were highly inconsistent. Furthermore, we found numerous other inconsistencies in reporting of sources of other bias in CSRs.\n\nConclusionCochrane authors mention a wide range of sources of other bias in the RoB tool and they inconsistently judge the same supporting explanations. Inconsistency in appraising risk of other bias hinders reliability and comparability of Cochrane systematic reviews. Furthermore, discrepant and erroneous judgments of bias in evidence synthesis will inevitably hinder implementation of evidence in routine clinical practice and reduce confidence of practitioners in otherwise trustworthy sources of information.

Explore related subjects

epidemiology Preprint

Keep this discovery

BibTeX RIS

Babic, A., Pijuk, A., Brazdilova, L., Georgieva, Y., Raposo Pereira, M. A., Poklepovic, T., Puljak, L.. 2018-07-11. Judgments of other bias in Cochrane systematic reviews of interventions are highly inconsistent and thus hindering use and comparability of evidence. https://doi.org/10.1101/366591

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related preprints

bioRxiv2018

Translating surveillance data into incidence estimates

Monitoring a population for a disease requires the hosts to be sampled and tested for the pathogen. This results in sampling series from which to estimate the disease incidence, i.e. the proportion of hosts infected. Existing estimation methods assume that disease incidence is not changing between monitoring rounds, resulting in underestimation of the disease incidence. In this paper we develop an incidence estimation model accounting for epidemic growth with monitoring rounds sampling varying incidence. We also show how to accommodate the asymptomatic period characteristic to most diseases. For practical use, we produce an approximation of the model, which is subsequently shown accurate for relevant epidemic and sampling parameters. Both the approximation and the full model are applied to stochastic spatial simulations of epidemics. The results prove their consistency for a very wide range of situations.

epidemiology↗

bioRxiv2018

The Swiss Primary Ciliary Dyskinesia registry: objectives, methods and first results

Primary Ciliary Dyskinesia (PCD) is a rare hereditary, multi-organ disease caused by defects in ciliary structure and function. It results in a wide range of clinical manifestations, most commonly in the upper and lower airways. Central data collection in national and international registries is essential to studying the epidemiology of rare diseases and filling in gaps in knowledge of diseases such as PCD. For this reason, the Swiss Primary Ciliary Dyskinesia Registry (CH-PCD) was founded in 2013 as a collaborative project between epidemiologists and adult and paediatric pulmonologists.\n\nThe registry records patients of any age, suffering from PCD, who are treated and resident in Switzerland. It collects information from patients identified through physicians, diagnostic facilities, and patient organisations. The registry dataset contains data on diagnostic evaluations, lung function, microbiology and imaging, symptoms, treatments, and hospitalizations.\n\nBy May 2018, CH-PCD has contacted 566 physicians of different specialties and identified 134 patients with PCD. At present this number represents an overall 1 in 63,000 prevalence of people diagnosed with PCD in Switzerland. Prevalence differs by age and region; it is highest in children and adults younger than 30 years, and in Espace Mittelland. The median age of patients in the registry is 25 years (range 5-73), and 49 patients have a definite PCD diagnosis based on recent international guidelines. Data from CH-PCD are contributed to international collaborative studies and the registry facilitates patient identification for nested studies.\n\nCH-PCD has proven to be a valuable research tool that already has highlighted weaknesses in PCD clinical practice in Switzerland. Development of centralised diagnostic and management centres and adherence to international guidelines are needed to improve diagnosis and management--particularly for adult PCD patients.

epidemiology↗

bioRxiv2018

Perfect Counterfactuals for Epidemic Simulations

Simulation studies are often used to predict the expected impact of control measures in infectious disease outbreaks. Typically, two independent sets of simulations are conducted, one with the intervetnion, and one without, and epidemic sizes (or some related metric) are compared to estimate the effect of the intervention. Since it is possible that controlled epidemics are larger than uncontrolled ones if there is substantial stochastic variation between epidemics, uncertainty intervals from this approach can include a negative effect even for an effective intervention. To more precisely estimate the number of cases an intervention will prevent within a single epidemic, here we develop a single world approach to matching simulations of controlled epidemics to their exact uncontrolled counterfac-tual. Our method borrows concepts from percolation approaches prune out possible epidemic histories and create potential epidemic graph that can be realized to create perfectly matched controlled and uncontrolled epidemics. We present an implementation of this method for a common class of compartmental models, and its application in a simple SIR model. Results illustrate how, at the cost of some computation time, this method substantially narrows confidence intervals and avoids non-sensical inferences.

epidemiology↗