Biggerstaff, M. | Search bioRxiv

bioRxiv2019

A systematic review and evaluation of Zika virus forecasting and prediction research during a public health emergency of international concern

INTRODUCTIONEpidemic forecasting and prediction tools have the potential to provide actionable information in the midst of emerging epidemics. While numerous predictive studies were published during the 2016-2017 Zika Virus (ZIKV) pandemic, it remains unknown how timely, reproducible and actionable the information produced by these studies was.\n\nMETHODSTo improve the functional use of mathematical modeling in support of future infectious disease outbreaks, we conducted a systematic review of all ZIKV prediction studies published during the recent ZIKV pandemic using the PRISMA guidelines. Using MEDLINE, EMBASE and grey literature review, we identified studies that forecasted, predicted or simulated ecological or epidemiological phenomenon related to the Zika pandemic that were published as of March 01, 2017. Eligible studies underwent evaluation of objectives, data sources, methods, timeliness, reproducibility, accessibility and clarity by independent reviewers.\n\nRESULTS2034 studies were identified, of which n = 73 met eligibility criteria. Spatial spread, R0 (basic reproductive number) and epidemic dynamics were most commonly predicted, with few studies predicting Guillain-Barre Syndrome burden (4%), sexual transmission risk (4%) and intervention impact (4%). Most studies specifically examined populations in the Americas (52%), with few African-specific studies (4%). Case count (67%), vector (41%) and demographic data (37%) were the most common data sources. Real-time internet data and pathogen genomic information were used in 7% and 0% of studies, respectively, and social science and behavioral data were typically absent in modeling efforts. Deterministic models were favored over stochastic approaches. Forty percent of studies made model data entirely available, 29% provided all relevant model code, 43% presented uncertainty in all predictions and 54% provided sufficient methodological detail allowing complete reproducibility. Fifty-one percent of predictions were published after the epidemic peak in the Americas. While the use of preprints improved the accessibility of ZIKV predictions by a median 119 days sooner than journal publication dates, they were used in only 30% of studies.\n\nCONCLUSIONSMany ZIKV predictions were published during the 2016-2017 pandemic. The accessibility, reproducibility, timeliness, and incorporation of uncertainty in these published predictions varied and indicates that there is substantial room for improvement. To enhance the utility of analytical tools for outbreak response, it is essential to improve the sharing of model data, code, and preprints for future outbreaks, epidemics and pandemics.\n\nAuthor summaryResearchers published many studies which sought to predict and forecast important features of Zika virus (ZIKV) infections and their spread during the 2016-2017 ZIKV pandemic. We conducted a comprehensive review of such ZIKV prediction studies and evaluated their aims, the data sources they used, which methods were used, how timely they were published, and whether they provided sufficient information to be used or reproduced by others. Of the 73 studies evaluated, we found that the accessibility, reproducibility, timeliness, and incorporation of uncertainty in these published predictions varied and indicates that there is substantial room for improvement. We identified that the release of study findings before formal journal publication ( pre-prints) increased the timeliness of Zika prediction studies, but note they were infrequently used during this public health emergency. Addressing these areas can improve our understanding of Zika and other outbreaks and ensure that forecasts can inform preparedness and response to future outbreaks, epidemics and pandemics.

epidemiology↗

bioRxiv2019

A Collaborative Multi-Model Ensemble for Real-Time Influenza Season Forecasting in the U.S.

Seasonal influenza results in substantial annual morbidity and mortality in the United States and worldwide. Accurate forecasts of key features of influenza epidemics, such as the timing and severity of the peak incidence in a given season, can inform public health response to outbreaks. As part of ongoing efforts to incorporate data and advanced analytical methods into public health decision-making, the United States Centers for Disease Control and Prevention (CDC) has organized seasonal influenza forecasting challenges since the 2013/2014 season. In the 2017/2018 season, 22 teams participated. A subset of four teams created a research consortium called the FluSight Network in early 2017. During the 2017/2018 season they worked together to produce a collaborative multi-model ensemble that combined 21 separate component models into a single model using a machine learning technique called stacking. This approach creates a weighted average of predictive densities where the weight for each component is based on that components forecast accuracy in past seasons. In the 2017/2018 influenza season, one of the largest seasonal outbreaks in the last 15 years, this multi-model ensemble performed better on average than all individual component models and placed second overall in the CDC challenge. It also outperformed the baseline multi-model ensemble created by the CDC that took a simple average of all models submitted to the forecasting challenge. This project shows that collaborative efforts between research teams to develop ensemble forecasting approaches can bring measurable improvements in forecast accuracy and important reductions in the variability of performance from year to year. Efforts such as this, that emphasize real-time testing and evaluation of forecasting models and facilitate the close collaboration between public health officials and modeling researchers, are essential to improving our understanding of how best to use forecasts to improve public health response to seasonal and emerging epidemic threats.

epidemiology↗

bioRxiv2018

Forecasting seasonal influenza in the U.S.: A collaborative multi-year, multi-model assessment of forecast performance

Influenza infects an estimated 9 to 35 million individuals each year in the United States and is a contributing cause for between 12,000 and 56,000 deaths annually. Seasonal outbreaks of influenza are common in temperate regions of the world, with highest incidence typically occurring in colder and drier months of the year. Real-time forecasts of influenza transmission can inform public health response to outbreaks. We present the results of a multi-institution collaborative effort to standardize the collection and evaluation of forecasting models for influenza in the US for the 2010/2011 through 2016/2017 influenza seasons. For these seven seasons, we assembled weekly real-time forecasts of 7 targets of public health interest from 22 different models. We compared forecast accuracy of each model relative to a historical baseline seasonal average. Across all regions of the US, over half of the models showed consistently better performance than the historical baseline when forecasting incidence of influenza-like illness 1, 2 and 3 weeks ahead of available data and when forecasting the timing and magnitude of the seasonal peak. In some regions, delays in data reporting were strongly and negatively associated with forecast accuracy. More timely reporting and an improved overall accessibility to novel and traditional data sources are needed to improve forecasting accuracy and its integration with real-time public health decision-making.

epidemiology↗