Predictive performance of multi-model ensemble forecasts of COVID-19 across European nations
Abstract
Background: Short-term forecasts of infectious disease contribute to situational awareness and capacity planning. Based on best practice in other fields and recent insights in infectious disease epidemiology, one can maximise forecasts’ predictive performance by combining independent models into an ensemble. Here we report the performance of ensemble predictions of COVID-19 cases and deaths across Europe from March 2021 to March 2022.
Methods: We created the European COVID-19 Forecast Hub, an online open-access platform where modellers upload weekly forecasts for 32 countries with results publicly visualised and evaluated. We created a weekly ensemble forecast from the equally-weighted average across individual models' predictive quantiles. We measured forecast accuracy using a baseline and relative Weighted Interval Score (rWIS). We retrospectively explored ensemble methods, including weighting by past performance.
Results: We collected weekly forecasts from 48 models, of which we evaluated 29 models alongside the ensemble model. The ensemble had a consistently strong performance across countries over time, performing better on rWIS than 91% of forecasts for deaths (N=763 predictions from 20 models), and 83% forecasts for cases (N=886 predictions from 23 models). Performance remained stable over a 4-week horizon for death forecasts but declined with longer horizons for cases. Among ensemble methods, the most influential choice came from using a median average instead of the mean, regardless of weighting component models.
Conclusions: Our results support combining independent models into an ensemble forecast to improve epidemiological predictions, and suggest that median averages yield better performance than methods based on means. We highlight that forecast consumers should place more weight on incident death forecasts than case forecasts at horizons greater than two weeks.
Funding: European Commission, Ministerio de Ciencia, Innovación y Universidades, FEDER; Agència de Qualitat i Avaluació Sanitàries de Catalunya; Netzwerk Universitätsmedizin; Health Protection Research Unit; Wellcome Trust; European Centre for Disease Prevention and Control; Ministry of Science and Higher Education of Poland; Federal Ministry of Education and Research; Los Alamos National Laboratory; German Free State of Saxony; NCBiR; FISR 2020 Covid-19 I Fase; Spanish Ministry of Health / REACT-UE (FEDER); National Institutes of General Medical Sciences; Ministerio de Sanidad/ISCIII; PERISCOPE European H2020; PERISCOPE European H2021; InPresa; National Institutes of Health, NSF, US Centers for Disease Control and Prevention, Google, University of Virginia, Defense Threat Reduction Agency.
Data availability
All source data were openly available before the study, originally available at: https://github.com/covid19-forecast-hub-europe/covid19-forecast-hub-europe. All data and code for this study are openly available on Github: covid19-forecast-hub-europe/euro-hub-ensemble.
Article and author information
Author details
Funding
NUM (Netzwerk Universitätsmedizin (NUM) project egePan (01KX2021))
- Jonas Dehning
- Sebastian Mohr
- Viola Priesemann
MUNI (Mathematical and Statistical modelling project (MUNI/A/1615/2020),MUNI/11/02202001/2020)
- Veronika Eclerová
- Lenka Pribylova
Ministerio de Sanidad/ISCIII
- Cesar Pérez Álvarez
- Borja Reina
- Jose L Aznarte
Ministry of Science and Higher Education of Poland (28/WFSN/2021)
- Rafal P Bartczuk
- Filip Dreger
- Magdalena Gruziel-Słomka
- Bartosz Krupa
- Antoni Moszyński
- Karol Niedzielewski
- Jedrzej Nowosielski
- Maciej Radwan
- Franciszek Rakowski
- Marcin Semeniuk
- Jakub Zieliński
- Jan Kisielewski
National Institutes of General Medical Sciences (R35GM119582)
- Graham Gibson
- Evan L Ray
- Nicholas G Reich
- Daniel Sheldon
- Yijin Wang
- Nutcha Wattanachit
FISR (SMIGE - Modelli statistici inferenziali per governare l'epidemia,FISR 2020 - Covid-19 I Fase,FISR2020IP_00156,Codice Progetto - PRJ-0695)
- Antonello Maruotti
- Gianfranco Lovison
- Alessio Farcomeni
AQuAS (Agència de Qualitat i Avaluació Sanitàries de Catalunya (AQuAS) through contract 2021_021OE)
- Inmaculada Villanueva
European Centre for Disease Prevention and Control
- Katharine Sherratt
- Hugo Gruson
European Commission (Communications Networks Content and Technology LC-01485746,Ministerio CIU/FEDER PGC2018-095456-B-I00)
- Sergio Alonso
- Enric Álvarez
- Daniel López
- Clara Prats
BMBF (Federal Ministry of Education and Research (BMBF; grant 05M18SIA))
- Stefan Heyder
- Thomas Hotz
- Jan Pablo Burgard
Health Protection Research Unit (NIHR200908)
- Nikos I Bosse
InPresa (Lombardy Region Italy)
- Fulvia Pennoni
- Francesco Bartolucci
LANL (Laboratory Directed Research and Development program of Los Alamos National Laboratory (LANL) under)
- Lauren Castro
- Geoffrey Fairchild
- Isaac Michaud
- Dave Osthus
The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.
Reviewing Editor
- Amy Wesolowski, Johns Hopkins Bloomberg School of Public Health, United States
Version history
- Preprint posted: June 16, 2022 (view preprint)
- Received: July 18, 2022
- Accepted: February 20, 2023
- Accepted Manuscript published: April 21, 2023 (version 1)
- Version of Record published: June 2, 2023 (version 2)
Copyright
This is an open-access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication.
Metrics
-
- 1,679
- views
-
- 257
- downloads
-
- 18
- citations
Views, downloads and citations are aggregated across all versions of this paper published by eLife.
Download links
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
Further reading
-
- Epidemiology and Global Health
We discuss 12 misperceptions, misstatements, or mistakes concerning the use of covariates in observational or nonrandomized research. Additionally, we offer advice to help investigators, editors, reviewers, and readers make more informed decisions about conducting and interpreting research where the influence of covariates may be at issue. We primarily address misperceptions in the context of statistical management of the covariates through various forms of modeling, although we also emphasize design and model or variable selection. Other approaches to addressing the effects of covariates, including matching, have logical extensions from what we discuss here but are not dwelled upon heavily. The misperceptions, misstatements, or mistakes we discuss include accurate representation of covariates, effects of measurement error, overreliance on covariate categorization, underestimation of power loss when controlling for covariates, misinterpretation of significance in statistical models, and misconceptions about confounding variables, selecting on a collider, and p value interpretations in covariate-inclusive analyses. This condensed overview serves to correct common errors and improve research quality in general and in nutrition research specifically.
-
- Ecology
- Epidemiology and Global Health
Zoonotic disease dynamics in wildlife hosts are rarely quantified at macroecological scales due to the lack of systematic surveys. Non-human primates (NHPs) host Plasmodium knowlesi, a zoonotic malaria of public health concern and the main barrier to malaria elimination in Southeast Asia. Understanding of regional P. knowlesi infection dynamics in wildlife is limited. Here, we systematically assemble reports of NHP P. knowlesi and investigate geographic determinants of prevalence in reservoir species. Meta-analysis of 6322 NHPs from 148 sites reveals that prevalence is heterogeneous across Southeast Asia, with low overall prevalence and high estimates for Malaysian Borneo. We find that regions exhibiting higher prevalence in NHPs overlap with human infection hotspots. In wildlife and humans, parasite transmission is linked to land conversion and fragmentation. By assembling remote sensing data and fitting statistical models to prevalence at multiple spatial scales, we identify novel relationships between P. knowlesi in NHPs and forest fragmentation. This suggests that higher prevalence may be contingent on habitat complexity, which would begin to explain observed geographic variation in parasite burden. These findings address critical gaps in understanding regional P. knowlesi epidemiology and indicate that prevalence in simian reservoirs may be a key spatial driver of human spillover risk.