Risk of bias in Cochrane systematic reviews: assessments of risk related to attrition bias are highly inconsistent

Babic, A.; Tokalic, R.; Silva Cunha, J. A.; Novak, I.; Suto, J.; Vidak, M.; Miosic, I.; Vuka, I.; Poklepovic Pericic, T.; Puljak, L.

doi:10.1101/366658

bioRxiv · 10.1101/366658

Risk of bias in Cochrane systematic reviews: assessments of risk related to attrition bias are highly inconsistent

Babic, A.·Tokalic, R.·Silva Cunha, J. A.·Novak, I.·Suto, J.·Vidak, M.·Miosic, I.·Vuka, I.·Poklepovic Pericic, T.·Puljak, L.

View original source ↗Export metadata ↓

Abstract

BackgroundAn important part of the systematic review methodology is appraisal of the risk of bias in included studies. Cochrane systematic reviews (CSRs) are considered golden standard regarding systematic review methodology, but Cochranes instructions for assessing risk of attrition bias are vague, which may lead to inconsistencies in authors assessments. The aim of this study was to analyze consistency of judgments and support for judgments of attrition bias in CSRs of interventions published in the Cochrane Database of Systematic Reviews (CDSR).\n\nMethodsWe analyzed CSRs published from July 2015 to June 2016 in the CDSR. We extracted data on number of included trials, judgment of attrition risk of bias for each included trial (low, unclear or high) and accompanying support for the judgment (supporting explanation). We also assessed how many CSRs had different judgments for the same supporting explanations.\n\nResultsIn the main analysis we included 10292 judgments and supporting explanations for attrition bias from 729 CSRs. We categorized supporting explanations for those judgments into four categories and we found that most of the supporting explanations were unclear. Numerical indicators for percent of attrition, as well as statistics related to attrition were judged very differently. One third of CSR authors had more than one category of supporting explanation; some had up to four different categories. Inconsistencies were found even with the number of judgments, names of risk of bias domains and different judgments for the same supporting explanations in the same CSR.\n\nConclusionWe found very high inconsistency in methods of appraising risk of attrition bias in recent Cochrane reviews. Systematic review authors need clear guidance about different categories they should assess and judgments for those explanations. Clear instructions about appraising risk of attrition bias will improve reliability of the Cochranes risk of bias tool, help authors in making decisions about risk of bias and help in making reliable decisions in healthcare.

Explore related subjects

epidemiology Preprint

Keep this discovery

BibTeX RIS

Babic, A., Tokalic, R., Silva Cunha, J. A., Novak, I., Suto, J., Vidak, M., Miosic, I., Vuka, I., Poklepovic Pericic, T., Puljak, L.. 2018-07-11. Risk of bias in Cochrane systematic reviews: assessments of risk related to attrition bias are highly inconsistent. https://doi.org/10.1101/366658

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related preprints

bioRxiv2018

Translating surveillance data into incidence estimates

Monitoring a population for a disease requires the hosts to be sampled and tested for the pathogen. This results in sampling series from which to estimate the disease incidence, i.e. the proportion of hosts infected. Existing estimation methods assume that disease incidence is not changing between monitoring rounds, resulting in underestimation of the disease incidence. In this paper we develop an incidence estimation model accounting for epidemic growth with monitoring rounds sampling varying incidence. We also show how to accommodate the asymptomatic period characteristic to most diseases. For practical use, we produce an approximation of the model, which is subsequently shown accurate for relevant epidemic and sampling parameters. Both the approximation and the full model are applied to stochastic spatial simulations of epidemics. The results prove their consistency for a very wide range of situations.

epidemiology↗

bioRxiv2018

The Swiss Primary Ciliary Dyskinesia registry: objectives, methods and first results

Primary Ciliary Dyskinesia (PCD) is a rare hereditary, multi-organ disease caused by defects in ciliary structure and function. It results in a wide range of clinical manifestations, most commonly in the upper and lower airways. Central data collection in national and international registries is essential to studying the epidemiology of rare diseases and filling in gaps in knowledge of diseases such as PCD. For this reason, the Swiss Primary Ciliary Dyskinesia Registry (CH-PCD) was founded in 2013 as a collaborative project between epidemiologists and adult and paediatric pulmonologists.\n\nThe registry records patients of any age, suffering from PCD, who are treated and resident in Switzerland. It collects information from patients identified through physicians, diagnostic facilities, and patient organisations. The registry dataset contains data on diagnostic evaluations, lung function, microbiology and imaging, symptoms, treatments, and hospitalizations.\n\nBy May 2018, CH-PCD has contacted 566 physicians of different specialties and identified 134 patients with PCD. At present this number represents an overall 1 in 63,000 prevalence of people diagnosed with PCD in Switzerland. Prevalence differs by age and region; it is highest in children and adults younger than 30 years, and in Espace Mittelland. The median age of patients in the registry is 25 years (range 5-73), and 49 patients have a definite PCD diagnosis based on recent international guidelines. Data from CH-PCD are contributed to international collaborative studies and the registry facilitates patient identification for nested studies.\n\nCH-PCD has proven to be a valuable research tool that already has highlighted weaknesses in PCD clinical practice in Switzerland. Development of centralised diagnostic and management centres and adherence to international guidelines are needed to improve diagnosis and management--particularly for adult PCD patients.

epidemiology↗

bioRxiv2018

Perfect Counterfactuals for Epidemic Simulations

Simulation studies are often used to predict the expected impact of control measures in infectious disease outbreaks. Typically, two independent sets of simulations are conducted, one with the intervetnion, and one without, and epidemic sizes (or some related metric) are compared to estimate the effect of the intervention. Since it is possible that controlled epidemics are larger than uncontrolled ones if there is substantial stochastic variation between epidemics, uncertainty intervals from this approach can include a negative effect even for an effective intervention. To more precisely estimate the number of cases an intervention will prevent within a single epidemic, here we develop a single world approach to matching simulations of controlled epidemics to their exact uncontrolled counterfac-tual. Our method borrows concepts from percolation approaches prune out possible epidemic histories and create potential epidemic graph that can be realized to create perfectly matched controlled and uncontrolled epidemics. We present an implementation of this method for a common class of compartmental models, and its application in a simple SIR model. Results illustrate how, at the cost of some computation time, this method substantially narrows confidence intervals and avoids non-sensical inferences.

epidemiology↗