Predictive performance of multi-model ensemble forecasts of COVID-19 across European nations
- London School of Hygiene and Tropical Medicine, United Kingdom
- European Centre for Disease Prevention and Control, Sweden
- Karlsruhe Institute of Technology, Germany
- Robert Koch Institute, Germany
- University of Massachusetts Amherst, United States
- Boston Children's Hospital, United States
- Charles University, Czech Republic
- École Polytechnique Fédérale de Lausanne, Switzerland
- Éducation Nationale, France
- Eidgenössische Technische Hochschule Zürich, Switzerland
- Forschungszentrum Jülich, Germany
- Frankfurt Institute for Advanced Studies, Germany
- Fraunhofer Institute for Industrial Mathematics, Germany
- Heidelberg University, Germany
- Helmholtz Centre for Infection Research, Germany
- IEM Inc, United States
- Independent, Austria
- Independent, United States
- Universitat Pompeu Fabra, Spain
- Institute of Computer Science, Czech Republic
- Institute of Information Theory and Automation, Czech Republic
- Inverence, Spain
- Los Alamos National Laboratory, United States
- LUMSA University, Italy
- Masaryk University, Czech Republic
- Massachusetts Institute of Technology, United States
- Max-Planck-Institut fur Dynamik und Selbstorganisation, Germany
- Max Planck Institute for Dynamics and Self-Organization, Germany
- Medical University of Gdansk, Poland
- Paul Scherrer Institute, Switzerland
- Politecnico di Milano, Italy
- Technical University of Kaiserlautern, Germany
- Technische Universitat Ilmenau, Germany
- Universidad Carlos III de Madrid, Spain
- Universidad Nacional de Educacion a Distancia, Spain
- Universitat de Barcelona, Spain
- Universitat Politecnica de Catalunya, Spain
- Universitat Trier, Germany
- University of Cologne, Germany
- University of Halle, Germany
- University of Ljubljana, Slovenia
- University of Milano-Bicocca, Italy
- University of Molise, Italy
- University of Oxford, United Kingdom
- University of Palermo, Italy
- University of Pavia, Italy
- University of Perugia, Italy
- University of Rome La Sapienza, Italy
- University of Rome Tor Vergata, Italy
- University of Southern California, United States
- University of Sydney, Australia
- University of Virginia, United States
- University of Warsaw, Poland
- University of Wrocla, Poland
- Universtat Leipzig, Germany
- Wroclaw University of Technology, Poland
- Wroclaw University of Science and Technology, Poland
Abstract
Background: Short-term forecasts of infectious disease contribute to situational awareness and capacity planning. Based on best practice in other fields and recent insights in infectious disease epidemiology, one can maximise forecasts’ predictive performance by combining independent models into an ensemble. Here we report the performance of ensemble predictions of COVID-19 cases and deaths across Europe from March 2021 to March 2022.
Methods: We created the European COVID-19 Forecast Hub, an online open-access platform where modellers upload weekly forecasts for 32 countries with results publicly visualised and evaluated. We created a weekly ensemble forecast from the equally-weighted average across individual models' predictive quantiles. We measured forecast accuracy using a baseline and relative Weighted Interval Score (rWIS). We retrospectively explored ensemble methods, including weighting by past performance.
Results: We collected weekly forecasts from 48 models, of which we evaluated 29 models alongside the ensemble model. The ensemble had a consistently strong performance across countries over time, performing better on rWIS than 91% of forecasts for deaths (N=763 predictions from 20 models), and 83% forecasts for cases (N=886 predictions from 23 models). Performance remained stable over a 4-week horizon for death forecasts but declined with longer horizons for cases. Among ensemble methods, the most influential choice came from using a median average instead of the mean, regardless of weighting component models.
Conclusions: Our results support combining independent models into an ensemble forecast to improve epidemiological predictions, and suggest that median averages yield better performance than methods based on means. We highlight that forecast consumers should place more weight on incident death forecasts than case forecasts at horizons greater than two weeks.
Funding: European Commission, Ministerio de Ciencia, Innovación y Universidades, FEDER; Agència de Qualitat i Avaluació Sanitàries de Catalunya; Netzwerk Universitätsmedizin; Health Protection Research Unit; Wellcome Trust; European Centre for Disease Prevention and Control; Ministry of Science and Higher Education of Poland; Federal Ministry of Education and Research; Los Alamos National Laboratory; German Free State of Saxony; NCBiR; FISR 2020 Covid-19 I Fase; Spanish Ministry of Health / REACT-UE (FEDER); National Institutes of General Medical Sciences; Ministerio de Sanidad/ISCIII; PERISCOPE European H2020; PERISCOPE European H2021; InPresa; National Institutes of Health, NSF, US Centers for Disease Control and Prevention, Google, University of Virginia, Defense Threat Reduction Agency.
Data availability
All source data were openly available before the study, originally available at: https://github.com/covid19-forecast-hub-europe/covid19-forecast-hub-europe. All data and code for this study are openly available on Github: covid19-forecast-hub-europe/euro-hub-ensemble.
Article and author information
Author details
Katharine Sherratt
Inmaculada Villanueva
Antonello Maruotti
Przemyslaw Porebski
Krzysztof Gogolewski
Funding
NUM (Netzwerk Universitätsmedizin (NUM) project egePan (01KX2021))
- Jonas Dehning
- Sebastian Mohr
- Viola Priesemann
MUNI (Mathematical and Statistical modelling project (MUNI/A/1615/2020),MUNI/11/02202001/2020)
- Veronika Eclerová
- Lenka Pribylova
Ministerio de Sanidad/ISCIII
- Cesar Pérez Álvarez
- Borja Reina
- Jose L Aznarte
Ministry of Science and Higher Education of Poland (28/WFSN/2021)
- Rafal P Bartczuk
- Filip Dreger
- Magdalena Gruziel-Słomka
- Bartosz Krupa
- Antoni Moszyński
- Karol Niedzielewski
- Jedrzej Nowosielski
- Maciej Radwan
- Franciszek Rakowski
- Marcin Semeniuk
- Jakub Zieliński
- Jan Kisielewski
National Institutes of General Medical Sciences (R35GM119582)
- Graham Gibson
- Evan L Ray
- Nicholas G Reich
- Daniel Sheldon
- Yijin Wang
- Nutcha Wattanachit
FISR (SMIGE - Modelli statistici inferenziali per governare l'epidemia,FISR 2020 - Covid-19 I Fase,FISR2020IP_00156,Codice Progetto - PRJ-0695)
- Antonello Maruotti
- Gianfranco Lovison
- Alessio Farcomeni
AQuAS (Agència de Qualitat i Avaluació Sanitàries de Catalunya (AQuAS) through contract 2021_021OE)
- Inmaculada Villanueva
European Centre for Disease Prevention and Control
- Katharine Sherratt
- Hugo Gruson
European Commission (Communications Networks Content and Technology LC-01485746,Ministerio CIU/FEDER PGC2018-095456-B-I00)
- Sergio Alonso
- Enric Álvarez
- Daniel López
- Clara Prats
BMBF (Federal Ministry of Education and Research (BMBF; grant 05M18SIA))
- Stefan Heyder
- Thomas Hotz
- Jan Pablo Burgard
Health Protection Research Unit (NIHR200908)
- Nikos I Bosse
InPresa (Lombardy Region Italy)
- Fulvia Pennoni
- Francesco Bartolucci
LANL (Laboratory Directed Research and Development program of Los Alamos National Laboratory (LANL) under)
- Lauren Castro
- Geoffrey Fairchild
- Isaac Michaud
- Dave Osthus
The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.
Copyright
This is an open-access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication.
Metrics
-
- 2,283
- views
-
- 327
- downloads
-
- 37
- citations
Views, downloads and citations are aggregated across all versions of this paper published by eLife.
Download links
A two-part list of links to download the article, or parts of the article, in various formats.
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
Predictive performance of multi-model ensemble forecasts of COVID-19 across European nations
eLife 12:e81916.
https://doi.org/10.7554/eLife.81916
Further reading
-
- Epidemiology and Global Health
- Microbiology and Infectious Disease
Background:
In many settings, a large fraction of the population has both been vaccinated against and infected by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). Hence, quantifying the protection provided by post-infection vaccination has become critical for policy. We aimed to estimate the protective effect against SARS-CoV-2 reinfection of an additional vaccine dose after an initial Omicron variant infection.
Methods:
We report a retrospective, population-based cohort study performed in Shanghai, China, using electronic databases with information on SARS-CoV-2 infections and vaccination history. We compared reinfection incidence by post-infection vaccination status in individuals initially infected during the April–May 2022 Omicron variant surge in Shanghai and who had been vaccinated before that period. Cox models were fit to estimate adjusted hazard ratios (aHRs).
Results:
275,896 individuals were diagnosed with real-time polymerase chain reaction-confirmed SARS-CoV-2 infection in April–May 2022; 199,312/275,896 were included in analyses on the effect of a post-infection vaccine dose. Post-infection vaccination provided protection against reinfection (aHR 0.82; 95% confidence interval 0.79–0.85). For patients who had received one, two, or three vaccine doses before their first infection, hazard ratios for the post-infection vaccination effect were 0.84 (0.76–0.93), 0.87 (0.83–0.90), and 0.96 (0.74–1.23), respectively. Post-infection vaccination within 30 and 90 days before the second Omicron wave provided different degrees of protection (in aHR): 0.51 (0.44–0.58) and 0.67 (0.61–0.74), respectively. Moreover, for all vaccine types, but to different extents, a post-infection dose given to individuals who were fully vaccinated before first infection was protective.
Conclusions:
In previously vaccinated and infected individuals, an additional vaccine dose provided protection against Omicron variant reinfection. These observations will inform future policy decisions on COVID-19 vaccination in China and other countries.
Funding:
This study was funded the Key Discipline Program of Pudong New Area Health System (PWZxk2022-25), the Development and Application of Intelligent Epidemic Surveillance and AI Analysis System (21002411400), the Shanghai Public Health System Construction (GWVI-11.2-XD08), the Shanghai Health Commission Key Disciplines (GWVI-11.1-02), the Shanghai Health Commission Clinical Research Program (20214Y0020), the Shanghai Natural Science Foundation (22ZR1414600), and the Shanghai Young Health Talents Program (2022YQ076).
-
- Epidemiology and Global Health
Background:
The role of circulating metabolites on child development is understudied. We investigated associations between children’s serum metabolome and early childhood development (ECD).
Methods:
Untargeted metabolomics was performed on serum samples of 5004 children aged 6–59 months, a subset of participants from the Brazilian National Survey on Child Nutrition (ENANI-2019). ECD was assessed using the Survey of Well-being of Young Children’s milestones questionnaire. The graded response model was used to estimate developmental age. Developmental quotient (DQ) was calculated as the developmental age divided by chronological age. Partial least square regression selected metabolites with a variable importance projection ≥1. The interaction between significant metabolites and the child’s age was tested.
Results:
Twenty-eight top-ranked metabolites were included in linear regression models adjusted for the child’s nutritional status, diet quality, and infant age. Cresol sulfate (β=–0.07; adjusted-p <0.001), hippuric acid (β=–0.06; adjusted-p <0.001), phenylacetylglutamine (β=–0.06; adjusted-p <0.001), and trimethylamine-N-oxide (β=–0.05; adjusted-p=0.002) showed inverse associations with DQ. We observed opposite directions in the association of DQ for creatinine (for children aged –1 SD: β=–0.05; pP=0.01;+1 SD: β=0.05; p=0.02) and methylhistidine (–1 SD: β = - 0.04; p=0.04;+1 SD: β=0.04; p=0.03).
Conclusions:
Serum biomarkers, including dietary and microbial-derived metabolites involved in the gut-brain axis, may potentially be used to track children at risk for developmental delays.
Funding:
Supported by the Brazilian Ministry of Health and the Brazilian National Research Council.
