Predictive performance of multi-model ensemble forecasts of COVID-19 across European nations

  1. Katharine Sherratt  Is a corresponding author
  2. Hugo Gruson
  3. Rok Grah
  4. Helen Johnson
  5. Rene Niehus
  6. Bastian Prasse
  7. Frank Sandmann
  8. Jannik Deuschel
  9. Daniel Wolffram
  10. Sam Abbott
  11. Alexander Ullrich
  12. Graham Gibson
  13. Evan L Ray
  14. Nicholas G Reich
  15. Daniel Sheldon
  16. Yijin Wang
  17. Nutcha Wattanachit
  18. Lijing Wang
  19. Jan Trnka
  20. Guillaume Obozinski
  21. Tao Sun
  22. Dorina Thanou
  23. Loic Pottier
  24. Ekaterina Krymova
  25. Jan H Meinke
  26. Maria Vittoria Barbarossa
  27. Neele Leithauser
  28. Jan Mohring
  29. Johanna Schneider
  30. Jaroslaw Wlazlo
  31. Jan Fuhrmann
  32. Berit Lange
  33. Isti Rodiah
  34. Prasith Baccam
  35. Heidi Gurung
  36. Steven Stage
  37. Bradley Suchoski
  38. Jozef Budzinski
  39. Robert Walraven
  40. Inmaculada Villanueva
  41. Vit Tucek
  42. Martin Smid
  43. Milan Zajicek
  44. Cesar Perez Alvarez
  45. Borja Reina
  46. Nikos I Bosse
  47. Sophie R Meakin
  48. Lauren Castro
  49. Geoffrey Fairchild
  50. Isaac Michaud
  51. Dave Osthus
  52. Pierfrancesco Alaimo Di Loro
  53. Antonello Maruotti
  54. Veronika Eclerova
  55. Andrea Kraus
  56. David Kraus
  57. Lenka Pribylova
  58. Bertsimas Dimitris
  59. Michael Lingzhi Li
  60. Soni Saksham
  61. Jonas Dehning
  62. Sebastian Mohr
  63. Viola Priesemann
  64. Grzegorz Redlarski
  65. Benjamin Bejar
  66. Giovanni Ardenghi
  67. Nicola Parolini
  68. Giovanni Ziarelli
  69. Wolfgang Bock
  70. Stefan Heyder
  71. Thomas Hotz
  72. David E Singh
  73. Miguel Guzman-Merino
  74. Jose L Aznarte
  75. David Morina
  76. Sergio Alonso
  77. Enric Alvarez
  78. Daniel Lopez
  79. Clara Prats
  80. Jan Pablo Burgard
  81. Arne Rodloff
  82. Tom Zimmermann
  83. Alexander Kuhlmann
  84. Janez Zibert
  85. Fulvia Pennoni
  86. Fabio Divino
  87. Marti Catala
  88. Gianfranco Lovison
  89. Paolo Giudici
  90. Barbara Tarantino
  91. Francesco Bartolucci
  92. Giovanna Jona Lasinio
  93. Marco Mingione
  94. Alessio Farcomeni
  95. Ajitesh Srivastava
  96. Pablo Montero-Manso
  97. Aniruddha Adiga
  98. Benjamin Hurt
  99. Bryan Lewis
  100. Madhav Marathe
  101. Przemyslaw Porebski
  102. Srinivasan Venkatramanan
  103. Rafal P Bartczuk
  104. Filip Dreger
  105. Anna Gambin
  106. Krzysztof Gogolewski
  107. Magdalena Gruziel-Slomka
  108. Bartosz Krupa
  109. Antoni Moszyński
  110. Karol Niedzielewski
  111. Jedrzej Nowosielski
  112. Maciej Radwan
  113. Franciszek Rakowski
  114. Marcin Semeniuk
  115. Ewa Szczurek
  116. Jakub Zielinski
  117. Jan Kisielewski
  118. Barbara Pabjan
  119. Kirsten Holger
  120. Yuri Kheifetz
  121. Markus Scholz
  122. Biecek Przemyslaw
  123. Marcin Bodych
  124. Maciej Filinski
  125. Radoslaw Idzikowski
  126. Tyll Krueger
  127. Tomasz Ozanski
  128. Johannes Bracher
  129. Sebastian Funk
  1. London School of Hygiene & Tropical Medicine, United Kingdom
  2. European Centre for Disease Prevention and Control (ECDC), Sweden
  3. Karlsruhe Institute of Technology, Germany
  4. Robert Koch Institute, Germany
  5. University of Massachusetts Amherst, United States
  6. Boston Children’s Hospital and Harvard Medical School, United States
  7. Third Faculty of Medicine, Charles University, Czech Republic
  8. Ecole Polytechnique Federale de Lausanne, Switzerland
  9. Éducation nationale, France
  10. Eidgenossische Technische Hochschule, Switzerland
  11. Forschungszentrum Jülich GmbH, Germany
  12. Frankfurt Institute for Advanced Studies, Germany
  13. Fraunhofer Institute for Industrial Mathematics, Germany
  14. Heidelberg University, Germany
  15. Helmholtz Centre for Infection Research, Germany
  16. IEM, Inc, United States
  17. Independent researcher, Austria
  18. Independent researcher, United States
  19. Institut d’Investigacions Biomèdiques August Pi i Sunyer, Universitat Pompeu Fabra, Spain
  20. Institute of Computer Science of the CAS, Czech Republic
  21. Institute of Information Theory and Automation of the CAS, Czech Republic
  22. Inverence, Spain
  23. Los Alamos National Laboratory, United States
  24. LUMSA University, Italy
  25. Masaryk University, Czech Republic
  26. Massachusetts Institute of Technology, United States
  27. Max-Planck-Institut für Dynamik und Selbstorganisation, Germany
  28. Medical University of Gdansk, Poland
  29. Paul Scherrer Institute, Switzerland
  30. Politecnico di Milano, Italy
  31. Technical University of Kaiserlautern, Germany
  32. Technische Universität Ilmenau, Germany
  33. Universidad Carlos III de Madrid, Spain
  34. Universidad Nacional de Educación a Distancia (UNED), Spain
  35. Universitat de Barcelona, Spain
  36. Universitat Politècnica de Catalunya, Spain
  37. Universitat Trier, Germany
  38. University of Cologne, Germany
  39. University of Halle, Germany
  40. University of Ljubljana, Slovenia
  41. University of Milano-Bicocca, Italy
  42. University of Molise, Italy
  43. University of Oxford, United Kingdom
  44. University of Palermo, Italy
  45. University of Pavia, Italy
  46. University of Perugia, Italy
  47. University of Rome "La Sapienza", Italy
  48. University of Rome "Tor Vergata", Italy
  49. University of Southern California, United States
  50. University of Sydney, Australia
  51. University of Virginia, United States
  52. University of Warsaw, Poland
  53. University of Bialystok, Poland
  54. University of Wroclaw, Poland
  55. Universtät Leipzig, Germany
  56. Warsaw University of Technology, Poland
  57. Wroclaw University of Science and Technology, Poland

Abstract

Background:

Short-term forecasts of infectious disease burden can contribute to situational awareness and aid capacity planning. Based on best practice in other fields and recent insights in infectious disease epidemiology, one can maximise the predictive performance of such forecasts if multiple models are combined into an ensemble. Here, we report on the performance of ensembles in predicting COVID-19 cases and deaths across Europe between 08 March 2021 and 07 March 2022.

Methods:

We used open-source tools to develop a public European COVID-19 Forecast Hub. We invited groups globally to contribute weekly forecasts for COVID-19 cases and deaths reported by a standardised source for 32 countries over the next 1–4 weeks. Teams submitted forecasts from March 2021 using standardised quantiles of the predictive distribution. Each week we created an ensemble forecast, where each predictive quantile was calculated as the equally-weighted average (initially the mean and then from 26th July the median) of all individual models’ predictive quantiles. We measured the performance of each model using the relative Weighted Interval Score (WIS), comparing models’ forecast accuracy relative to all other models. We retrospectively explored alternative methods for ensemble forecasts, including weighted averages based on models’ past predictive performance.

Results:

Over 52 weeks, we collected forecasts from 48 unique models. We evaluated 29 models’ forecast scores in comparison to the ensemble model. We found a weekly ensemble had a consistently strong performance across countries over time. Across all horizons and locations, the ensemble performed better on relative WIS than 83% of participating models’ forecasts of incident cases (with a total N=886 predictions from 23 unique models), and 91% of participating models’ forecasts of deaths (N=763 predictions from 20 models). Across a 1–4 week time horizon, ensemble performance declined with longer forecast periods when forecasting cases, but remained stable over 4 weeks for incident death forecasts. In every forecast across 32 countries, the ensemble outperformed most contributing models when forecasting either cases or deaths, frequently outperforming all of its individual component models. Among several choices of ensemble methods we found that the most influential and best choice was to use a median average of models instead of using the mean, regardless of methods of weighting component forecast models.

Conclusions:

Our results support the use of combining forecasts from individual models into an ensemble in order to improve predictive performance across epidemiological targets and populations during infectious disease epidemics. Our findings further suggest that median ensemble methods yield better predictive performance more than ones based on means. Our findings also highlight that forecast consumers should place more weight on incident death forecasts than incident case forecasts at forecast horizons greater than 2 weeks.

Funding:

AA, BH, BL, LWa, MMa, PP, SV funded by National Institutes of Health (NIH) Grant 1R01GM109718, NSF BIG DATA Grant IIS-1633028, NSF Grant No.: OAC-1916805, NSF Expeditions in Computing Grant CCF-1918656, CCF-1917819, NSF RAPID CNS-2028004, NSF RAPID OAC-2027541, US Centers for Disease Control and Prevention 75D30119C05935, a grant from Google, University of Virginia Strategic Investment Fund award number SIF160, Defense Threat Reduction Agency (DTRA) under Contract No. HDTRA1-19-D-0007, and respectively Virginia Dept of Health Grant VDH-21-501-0141, VDH-21-501-0143, VDH-21-501-0147, VDH-21-501-0145, VDH-21-501-0146, VDH-21-501-0142, VDH-21-501-0148. AF, AMa, GL funded by SMIGE - Modelli statistici inferenziali per governare l'epidemia, FISR 2020-Covid-19 I Fase, FISR2020IP-00156, Codice Progetto: PRJ-0695. AM, BK, FD, FR, JK, JN, JZ, KN, MG, MR, MS, RB funded by Ministry of Science and Higher Education of Poland with grant 28/WFSN/2021 to the University of Warsaw. BRe, CPe, JLAz funded by Ministerio de Sanidad/ISCIII. BT, PG funded by PERISCOPE European H2020 project, contract number 101016233. CP, DL, EA, MC, SA funded by European Commission - Directorate-General for Communications Networks, Content and Technology through the contract LC-01485746, and Ministerio de Ciencia, Innovacion y Universidades and FEDER, with the project PGC2018-095456-B-I00. DE., MGu funded by Spanish Ministry of Health / REACT-UE (FEDER). DO, GF, IMi, LC funded by Laboratory Directed Research and Development program of Los Alamos National Laboratory (LANL) under project number 20200700ER. DS, ELR, GG, NGR, NW, YW funded by National Institutes of General Medical Sciences (R35GM119582; the content is solely the responsibility of the authors and does not necessarily represent the official views of NIGMS or the National Institutes of Health). FB, FP funded by InPresa, Lombardy Region, Italy. HG, KS funded by European Centre for Disease Prevention and Control. IV funded by Agencia de Qualitat i Avaluacio Sanitaries de Catalunya (AQuAS) through contract 2021-021OE. JDe, SMo, VP funded by Netzwerk Universitatsmedizin (NUM) project egePan (01KX2021). JPB, SH, TH funded by Federal Ministry of Education and Research (BMBF; grant 05M18SIA). KH, MSc, YKh funded by Project SaxoCOV, funded by the German Free State of Saxony. Presentation of data, model results and simulations also funded by the NFDI4Health Task Force COVID-19 (https://www.nfdi4health.de/task-force-covid-19-2) within the framework of a DFG-project (LO-342/17-1). LP, VE funded by Mathematical and Statistical modelling project (MUNI/A/1615/2020), Online platform for real-time monitoring, analysis and management of epidemic situations (MUNI/11/02202001/2020); VE also supported by RECETOX research infrastructure (Ministry of Education, Youth and Sports of the Czech Republic: LM2018121), the CETOCOEN EXCELLENCE (CZ.02.1.01/0.0/0.0/17-043/0009632), RECETOX RI project (CZ.02.1.01/0.0/0.0/16-013/0001761). NIB funded by Health Protection Research Unit (grant code NIHR200908). SAb, SF funded by Wellcome Trust (210758/Z/18/Z).

Editor's evaluation

This large-scale collaborative study is a timely contribution that will be of interest to researchers working in the fields of infectious disease forecasting and epidemic control. This paper provides a comprehensive evaluation of the predictive skills of real-time COVID-19 forecasting models in Europe. The conclusions of the paper are well supported by the data and are consistent with findings from studies in other countries.

https://doi.org/10.7554/eLife.81916.sa0

Introduction

Epidemiological forecasts make quantitative statements about a disease outcome in the near future. Forecasting targets can include measures of prevalent or incident disease and its severity, for some population over a specified time horizon. Researchers, policy makers, and the general public have used such forecasts to understand and respond to the global outbreaks of COVID-19 (Van Basshuysen et al., 2021; CDC, 2020; European Centre for Disease Prevention and Control, 2021c). At the same time, forecasters use a variety of methods and models for creating and publishing forecasts, varying in both defining the forecast outcome and in reporting the probability distribution of outcomes (Zelner et al., 2021; James et al., 2021).

Within Europe, comparing forecasts across both models and countries can support a range of national policy needs simultaneously. European public health professionals operate across national, regional, and continental scales, with strong existing policy networks in addition to rich patterns of cross-border migration influencing epidemic dynamics. A majority of European countries also cooperate in setting policy with inter-governmental European bodies such as the European Centre for Disease Prevention and Control (ECDC). In this case, a consistent approach to forecasting across the continent as a whole can support accurately informing cross-European monitoring, analysis, and guidance (European Centre for Disease Prevention and Control, 2021c). At a regional level, multi-country forecasts can support a better understanding of the impact of regional migration networks. Meanwhile, where there is limited capacity for infectious disease forecasting at a national level, forecasters generating multi-country results can provide an otherwise-unavailable opportunity for forecasts to inform national situational awareness. Some independent forecasting models have sought to address this by producing multi-country results (Aguas et al., 2020; Adib et al., 2021; Agosto and Giudici, 2020; Agosto et al., 2021).

Variation in forecast methods and presentation makes it difficult to compare predictive performance between forecast models, and from there to derive objective arguments for using one forecast over another. This confounds the selection of a single representative forecast and reduces the reliability of the evidence base for decisions based on forecasts. A ‘forecast hub’ is a centralised effort to improve the transparency and usefulness of forecasts, by standardising and collating the work of many independent teams producing forecasts (Reich et al., 2019a). A hub sets a commonly agreed-upon structure for forecast targets, such as type of disease event, spatio-temporal units, or the set of quantiles of the probability distribution to include from probabilistic forecasts. For instance, a hub may collect predictions of the total number of cases reported in a given country for each day in the next 2 weeks. Forecasters can adopt this format and contribute forecasts for centralised storage in the public domain.

This shared infrastructure allows forecasts produced from diverse teams and methods to be visualised and quantitatively compared on a like-for-like basis, which can strengthen public and policy use of disease forecasts. The underlying approach to creating a forecast hub was pioneered in climate modelling and adapted for collaborative epidemiological forecasts of dengue (Johansson et al., 2019) and influenza in the USA (Reich et al., 2019a; Reich et al., 2019b). This infrastructure was adapted for forecasts of short-term COVID-19 cases and deaths in the US (Cramer et al., 2021a; Ray et al., 2020), prompting similar efforts in some European countries (Bracher et al., 2021c; Funk et al., 2020; Bicher et al., 2020).

Standardising forecasts allows for combining multiple forecasts into a single ensemble with the potential for an improved predictive performance. Evidence from previous efforts in multi-model infectious disease forecasting suggests that forecasts from an ensemble of models can be consistently high performing compared to any one of the component models (Johansson et al., 2019; Reich et al., 2019b; Viboud et al., 2018). Elsewhere, weather forecasting has a long-standing use of building ensembles of models using diverse methods with standardised data and formatting in order to improve performance (Buizza, 2019; Moran et al., 2016).

The European COVID-19 Forecast Hub (European Covid-19 Forecast Hub, 2023d) is a project to collate short-term forecasts of COVID-19 across 32 countries in the European region. The Hub is funded and supported by the ECDC, with the primary aim to provide reliable information about the near-term epidemiology of the COVID-19 pandemic to the research and policy communities and the general public (European Centre for Disease Prevention and Control, 2021c). Second, the Hub aims to create infrastructure for storing and analysing epidemiological forecasts made in real time by diverse research teams and methods across Europe. Third, the Hub aims to maintain a community of infectious disease modellers underpinned by open science principles.

We started formally collating and combining contributions to the European Forecast Hub in March 2021. Here, we investigate the predictive performance of an ensemble of all forecasts contributed to the Hub in real time each week, as well as the performance of variations of ensemble methods created retrospectively.

Materials and methods

We developed infrastructure to host and analyse prospective forecasts of COVID-19 cases and deaths. The infrastructure is compatible with equivalent research software from the US (Cramer et al., 2021c; Wang et al., 2021) and German and Polish COVID-19 (Bracher et al., 2020) Forecast Hubs, and easy to replicate for new forecasting collaborations.

Forecast targets and models

Request a detailed protocol

We sought forecasts for the incidence of COVID-19 as the total reported number of cases and deaths per week. We considered forecasts for 32 countries in Europe, including all countries of the European Union, European Free Trade Area, and the United Kingdom. We compared forecasts against observed data reported for each country by Johns Hopkins University (JHU, Dong et al., 2020). JHU data sources included a mix of national and aggregated subnational data. We aggregated incidence over the Morbidity and Mortality Weekly Report (MMWR) epidemiological week definition of Sunday through Saturday.

Teams could express their uncertainty around any single forecast target by submitting predictions for up to 23 quantiles (from 0.01 to 0.99) of the predictive probability distribution. Teams could also submit a single point forecast. At the first submission, we asked teams to add a pre-specified set of metadata briefly describing the forecasting team and methods (provided online and in supplementary information). No restrictions were placed on who could submit forecasts. To increase participation, we actively contacted known forecasting teams across Europe and the US and advertised among the ECDC network. Teams submitted a broad spectrum of model types, ranging from mechanistic to empirical models, agent-based and statistical models, and ensembles of multiple quantitative or qualitative models (described at European Covid-19 Forecast Hub, 2023a). We maintain a full project specification with a detailed submissions protocol (European Covid-19 Forecast Hub, 2023c).

We collected forecasts submitted weekly in real time over the 52-week period from 08 March 2021 to 07 March 2022. Teams submitted at latest 2 days after the complete dataset for the latest forecasting week became available each Sunday. We implemented an automated validation programme to check that each new forecast conformed to standardised formatting. Forecast validation ensured a monotonic increase of predictions with each increasing quantile, integer-valued non-negative counts of predicted cases, as well as consistent date and location definitions.

Each week we used all available valid forecasts to create a weekly real-time ensemble model (referred to as ‘the ensemble’ from here on), for each of the 256 possible forecast targets: incident cases and deaths in 32 locations over the following one through 4 weeks. The ensemble method was an unweighted average of all models’ forecast values, at each predictive quantile for a given location, target, and horizon. From 08 March 2021, we used the arithmetic mean. However we noticed that including highly anomalous forecasts in a mean ensemble produced extremely wide uncertainty. To mitigate this, from 26th July 2021 onwards the ensemble instead used a median of all predictive quantiles.

We created an open and publicly accessible interface to the forecasts and ensemble, including an online visualisation tool allowing viewers to see past data and interact with one or multiple forecasts for each country and target for up to 4 weeks’ horizon (European Covid-19 Forecast Hub, 2023b). All forecasts, metadata, and evaluations are freely available and held on Github (European Covid-19 Forecast Hub, 2023d) (archived in real-time at Sherratt, 2022), and Zoltar, a platform for hosting epidemiological forecasts (EpiForecasts, 2021; Reich et al., 2021). In the codebase for this study (covid19-forecast-hub-europe, 2022) we provide a simple method and instructions for downloading and preparing these data for analysis using R. We encourage other researchers to freely use and adapt this to support their own analyses.

Forecast evaluation

Request a detailed protocol

In this study, we focused only on the comparative performance of forecasting models relative to each other. Performance in absolute terms is available on the Hub website (European Covid-19 Forecast Hub, 2023b). For each model, we assessed calibration and overall predictive performance. We evaluated all previous forecasts against actual observed values for each model, stratified by the forecast horizon, location, and target. We calculated scores using the scoringutils R package (Bosse et al., 2023). We removed any forecast surrounding (both the week of, and the first week after) a strongly anomalous data point. We defined anomalous as where any subsequent data release revised that data point by over 5%.

To investigate calibration, we assessed coverage as the correspondence between the forecast probability of an event and the observed frequency of that event. This usage follows previous work in epidemic forecasting (Bracher et al., 2021a), and is related to the concept of reliability for binary forecasts. We established the accuracy of each model’s prediction boundaries as the coverage of the predictive intervals. We calculated coverage at a given interval level k, where k[0,1], as the proportion p of observations that fell within the corresponding central predictive intervals across locations and forecast dates. A perfectly calibrated model would have p=k at all 11 levels (corresponding to 22 quantiles excluding the median). An underconfident model at level k would have p>k, i.e. more observations fall within a given interval than expected. In contrast, an overconfident model at level k would have p<k, i.e. fewer observations fall within a given interval than expected. We here focus on coverage at the k=0.5 and k=0.95 levels.

We also assessed the overall predictive performance of weekly forecasts using the Weighted Interval Score~(WIS) across all available quantiles. The WIS represents a parsimonious approach to scoring forecasts based on uncertainty represented as forecast values across a set of quantiles (Bracher et al., 2021a), and is a strictly proper scoring rule, that is, it is optimal for predictions that come from the data-generating model. As a consequence, the WIS encourages forecasters to report predictions representing their true belief about the future (Gneiting and Raftery, 2007). Each forecast for a given location and date is scored based on an observed count of weekly incidence, the median of the predictive distribution and the predictive upper and lower quantiles corresponding to the central predictive interval level.

Not all models provided forecasts for all locations and dates, and we needed to compare predictive performance in the face of various levels of missingness across each forecast target. Therefore we calculated a relative WIS. This is a measure of forecast performance which takes into account that different teams may not cover the same set of forecast targets (i.e. weeks and locations). The relative WIS is computed using a pairwise comparison tournament where for each pair of models a mean score ratio is computed based on the set of shared targets. The relative WIS of a model with respect to another model is then the ratio of their respective geometric mean of the mean score ratios, such that smaller values indicate better performance.

We scaled the relative WIS of each model with the relative WIS of a baseline model, for each forecast target, location, date, and horizon. The baseline model assumes case or death counts stay the same as the latest data point over all future horizons, with expanding uncertainty, described previously in Cramer et al., 2021b. In this study, we report the relative WIS of each model with respect to the baseline model.

Retrospective ensemble methods

Request a detailed protocol

We retrospectively explored alternative methods for combining forecasts for each target at each week. A natural way to combine probability distributions available in the quantile format Genest, 1992 used here is

F1(α)=i=1nwiFi1(α),

Where F1Fn are the cumulative distribution functions of the individual probability distributions (in our case, the predictive distributions of each forecast model i contributed to the hub), wi are a set of weights in [0,1]; and α are the quantile levels, such that following notation introduced in Genest, 1992,

F1(α)=inf{t:Fi(t)α}.

Different ensemble choices then mainly translate to the choice of weights wi. An arithmetic mean ensemble uses weights at wi=1/n, where all weights are equal and sum up to 1.

Alternatively, we can choose a set of weights to apply to forecasts before they are combined. Numerous options exist for choosing these weights with the aim to maximise predictive performance, including choosing weights to reflect each forecast’s past performance (thereby moving from an untrained to a trained ensemble). A straightforward choice is so-called inverse score weighting. In this case, the weights are calculated as

wi=1Si,

where Si reflects the forecasting skill calculated as the relative WIS of forecaster i, calculated over all available model data, and normalised so that weights sum to 1. This method of weighting was found in the US to outperform unweighted scores during some time periods (Taylor and Taylor, 2023) but this was not confirmed in a similar study in Germany and Poland (Bracher et al., 2021c).

When constructing ensembles from quantile means, a single outlier can have an oversized effect on the ensemble forecast. Previous research has found that a median ensemble, replacing the arithmetic mean of each quantile with a median of the same values, yields competitive performance while maintaining robustness to outlying forecasts (Ray et al., 2022). Building on this, we also created weighted median ensembles using the weights described above and a Harrel-Davis quantile estimator with a beta function to approximate the weighted percentiles (Harrell and Davis, 1982). We then compared the performance of unweighted and inverse relative WIS weighted mean and median ensembles, comparing the ratio of interval scores between each ensemble model relative to the baseline model.

Results

For 32 European countries, we collected, visualised, and made available online weekly COVID-19 forecasts and observed data (Sherratt, 2022). Over the whole study period, we collected forecasts from 48 unique models. Modellers created forecasts choosing from a set of 32 possible locations, four time horizons, and two variables, and modellers variously joined and left the Hub over time. This meant the number of models contributing to the Hub varied over time and by forecasting target. Using all models and the ensemble, we created 2139 forecasting scores, where each score summarises a unique combination of forecasting model, variable, country, and week ahead horizon (Figure 1).

Total number of forecasts included in evaluation, by target location, week ahead horizon, and variable.

Of the total 48 models, we received the most forecasts for Germany, with 29 unique models submitting 1-week case forecasts, while only 12 models ever submitted 4-week case or death forecasts for Liechtenstein. Modelling teams also differed in how they expressed uncertainty. Only three models provided point forecasts with no estimate of uncertainty around their predictions, while 41 models provided the full set of 23 probabilistic quantiles across the predictive distribution for each target.

In this evaluation we included 29 models in comparison to the ensemble forecast (Figure 1). We have included metadata provided by modellers in the supplement and online (Sherratt, 2022). In this evaluation, at most 15 models contributed forecasts for cases in Germany at the 1 week horizon, with an accumulated 592 forecast scores for that single target over the study period. In contrast, deaths in Finland at the 2 week horizon saw the smallest number of forecasts, with only 6 independent models contributing 24 forecast scores at any time over the 52-week period. Of the 29 models included in this evaluation, 5 models provided less than the full set of 23 quantiles, and were excluded when creating the ensemble. No ensemble forecast was composed of less than 3 independent models.

We visually compared the absolute performance of forecasts in predicting numbers of incident cases and deaths. We observed that forecasts predicted well in times of stable epidemic behaviour, while struggling to accurately predict at longer horizons around inflection points, for example during rapid changes in population-level behaviour or surveillance. Forecast models varied widely in their ability to predict and account for the introduction of new variants, giving the ensemble forecast over these periods a high level of uncertainty. An example of weekly forecasts from the ensemble model is shown in Figure 2.

Ensemble forecasts of weekly incident cases in Germany over periods of increasing SARS-CoV-2 variants Delta (B.1.617.2, left) and Omicron (B.1.1.529, right).

Black indicates observed data. Coloured ribbons represent each weekly forecast of 1–4 weeks ahead (showing median, 50%, and 90% probability). For each variant, forecasts are shown over an x-axis bounded by the earliest dates at which 5% and 99% of sequenced cases were identified as the respective variant of concern, while vertical dotted lines indicate the approximate date that the variant reached dominance (>50% sequenced cases).

In relative terms, the ensemble of all models performed well compared to both its component models and the baseline. By relative WIS scaled against a baseline of 1 (where a score <1 indicates outperforming the baseline), the median score of forecasts from the Hub ensemble model was 0.71, within an interquartile range of 0.61 at 25% probability to 0.88 at 75% probability. Meanwhile the median score of forecasts across all participating models (excluding the Hub ensemble) was 1.04 (IQR 0.82–1.36).

Across all horizons and locations, the ensemble performed better on scaled relative WIS than 83% of forecast scores when forecasting cases (with a total N=886 from 23 unique models), and 91% of scores for forecasts of incident deaths (N=763 scores from 20 models). We also saw high performance from the ensemble when evaluating against all models including those who did not submit the full set of probabilistic quantile predictions (80% for cases with N=1006 scores from 28 models, and 88% for deaths, N=877 scores from 24 models).

The performance of individual and ensemble forecasts varied by length of the forecast horizon (Figure 3). At each horizon, the typical performance of the ensemble outperformed both the baseline model and the aggregated scores of all its component models, although we saw wide variation between individual models in performance across horizons. Both individual models and the ensemble saw a trend of worsening performance at longer horizons when forecasting cases with the median scaled relative WIS of the ensemble across locations worsened from 0.62 for 1-week ahead forecasts to 0.9 when forecasting 4 weeks ahead. Performance for forecasts of deaths was more stable over one through 4 weeks, with median ensemble performance moving from 0.69 to 0.76 across the 4-week horizons.

Performance of short-term forecasts aggregated across all individually submitted models and the Hub ensemble, by horizon, forecasting cases (left) and deaths (right).

Performance measured by relative weighted interval score scaled against a baseline (dotted line, 1), and coverage of uncertainty at the 50% and 95% levels. Boxplot, with width proportional to number of observations, show interquartile ranges with outlying scores as faded points. The target range for each set of scores is shaded in yellow.

We observed similar trends in performance across horizon when considering how well the ensemble was calibrated with respect to the observed data. At 1 week ahead the case ensemble was well calibrated (ca. 50% and 95% nominal coverage at the 50% and 95% levels, respectively). This did not hold at longer forecast horizons as the case forecasts became increasingly over-confident. Meanwhile, the ensemble of death forecasts was well calibrated at the 95% level across all horizons, and the calibration of death forecasts at the 50% level improved with lengthening horizons compared to being underconfident at shorter horizons.

The ensemble also performed consistently well in comparison to individual models when forecasting across countries (Figure 4). In total, across 32 countries forecasting for 1 through 4 weeks, when forecasting cases the ensemble outperformed 75% of component models in 22 countries, and outperformed all available models in 3 countries. When forecasting deaths, the ensemble outperformed 75% and 100% of models in 30 and 8 countries, respectively. Considering only the the 2-week horizon shown in Figure 4, the ensemble of case forecasts outperformed 75% models in 25 countries and all models in only 12 countries. At the 2-week horizon for forecasts of deaths, the ensemble outperformed 75% and 100% of its component models in 30 and 26 countries, respectively.

Performance of short-term forecasts across models and median ensemble (asterisk), by country, forecasting cases (top) and deaths (bottom) for 2-week ahead forecasts, according to the relative weighted interval score.

Boxplots show interquartile ranges, with outliers as faded points, and the ensemble model performance is marked by an asterisk. y-axis is cut-off to an upper bound of 4 for readability.

We considered alternative methods for creating ensembles from the participating forecasts, using either a mean or median to combine either weighted or unweighted forecasts. We evaluated each alternative ensemble model against the baseline model, taking the mean score ratio across all targets (Table 1). Across locations we observed that the median outperformed the mean across all one through 4 week horizons and both cases and death targets, for all but cases at the 1 week horizon. This held regardless of whether the component forecasts were weighted or unweighted by their individual past performance. Between methods of combination, weighting made little difference to the performance of the median ensemble, but appeared to improve performance of a mean ensemble in forecasting deaths.

Table 1
Predictive performance of main ensembles, as measured by the mean ratio of interval scores against the baseline ensemble.
HorizonWeighted meanWeighted medianUnweighted meanUnweighted median
Cases
1 week0.630.640.610.64
2 weeks0.720.710.690.69
3 weeks0.820.760.820.72
4 weeks1.070.861.120.78
Deaths
1 week0.650.611.810.61
2 weeks0.580.541.290.54
3 weeks0.640.571.170.53
4 weeks0.820.670.840.62

Discussion

We collated 12 months of forecasts of COVID-19 cases and deaths across 32 countries in Europe, collecting from multiple independent teams and using a principled approach to standardising both forecast targets and the predictive distribution of forecasts. We combined these into an ensemble forecast and compared the relative performance of forecasts between models, finding that the ensemble forecasts outperformed most individual models across all countries and horizons over time.

Across all models we observed that forecasting changes in trend in real time was particularly challenging. Our study period included multiple fundamental changes in viral-, individual-, and population-level factors driving the transmission of COVID-19 across Europe. In early 2021, the introduction of vaccination started to change population-level associations between infections, cases, and deaths (European Centre for Disease Prevention and Control, 2021b), while the Delta variant emerged and became dominant (European Centre for Disease Prevention and Control, 2021a). Similarly from late 2021 we saw the interaction of individually waning immunity during the emergence and global spread of the Omicron variant (European Centre for Disease Prevention and Control, 2022b). Neither the extent nor timing of these factors were uniform across European countries covered by the Forecast Hub (European Centre for Disease Prevention and Control, 2023). This meant that the performance of any single forecasting model depended partly on the ability, speed, and precision with which it could adapt to new conditions for each forecast target.

We observed a contrast between a more stable performance of forecasting deaths further into the future compared to forecasts of cases. Previous work has found rapidly declining performance for case forecasts with increasing horizon (Cramer et al., 2021b; Castro et al., 2020), while death forecasts can perform well with up to 6 weeks lead time (Friedman et al., 2021). We can link this to the specific epidemic dynamics in this study.

First, COVID-19 has a typical serial interval of less than a week (Alene et al., 2021). This implies that case forecasts of more than 2 weeks only remain valid if rates of both transmission and detection remain stable over the entire forecast horizon. In contrast, we saw rapid changes in epidemic dynamics across many countries in Europe over our study period, impacting the longer term case forecasts.

Second, we can interpret the higher reliability of death forecasts as due to the different lengths and distributions of time lags from infection to case and death reporting (Jin, 2021). For example, a spike in infections may be matched by a consistently sharp increase in case reporting, but a longer tailed distribution of the subsequent increase in death reports. This creates a lower magnitude of fluctuation in the time-series of deaths compared to that of cases. Similarly, surveillance data for death reporting is substantially more consistent, with fewer errors and retrospective corrections, than case reporting (Català et al., 2021).

Third, we also note that the performance of trend-based forecasts may have benefited from the slower changes to trends in incident deaths caused by gradually increasing vaccination rates. These features allow forecasters to incorporate the effect of changes in transmission more easily when forecasting deaths, compared to cases.

We found the ensemble in this study continued to outperform both other models and the baseline at up to 4 weeks ahead. Our results support previous findings that ensemble forecasts are the best or nearly the best performing models with respect to absolute predictive performance and appropriate coverage of uncertainty (Funk et al., 2020; Viboud et al., 2018; Cramer et al., 2021b). While the ensemble was consistently high performing, it was not strictly dominant across all forecast targets, reflecting findings from previous comparable studies of COVID-19 forecasts (Bracher et al., 2021c; Brooks, 2020). Our finding suggests the usefulness of an ensemble as a robust summary when forecasting across many spatio-temporal targets, without replacing the importance of communicating the full range of model predictions.

When exploring variations in ensemble methods, we found that the choice of median over means yielded the most consistent improvement in predictive performance, regardless of the method of weighting. Other work has supported the importance of the median in providing a stable forecast that better accounts for outlier forecasts than the mean (Brooks, 2020), although this finding may be dependent on the quality of the individual forecast submissions. In contrast, weighing models by past performance did not result in any consistent improvement in performance. This is in line with existing mixed evidence for any optimal ensemble method for combining short term probabilistic infectious disease forecasts. Many methods of combination have performed competitively in analyses of forecasts for COVID-19 in the US, including the simple mean and weighted approaches outperforming unweighted or median methods (Taylor and Taylor, 2023). This contrasts with later analyses finding weighted methods to give similar performance to a median average (Ray et al., 2020; Brooks, 2020). We can partly explain this inconsistency if performance of each method depends on the outcome being predicted (cases, deaths), its count (incident, cumulative) and absolute level, the changing disease dynamics, and the varying quality and quantity of forecasting teams over time.

We note several limitations in our approach to assessing the relative performance of an ensemble among forecast models. While we have described differences in model scores, we have not used any formal statistical test for comparing forecast scores, such as the Diebold-Mariano test (Diebold and Mariano, 1995), recognising that it is unclear how this is best achieved across many models. Our results are the outcome of evaluating forecasts against a specific performance metric and baseline, where multiple options for evaluation exist and the choice reflects the aim of the evaluation process. Further, our choice of baseline model affects the given performance scores in absolute terms, and more generally the choice of appropriate baseline for epidemic forecast models is not obvious when assessing infectious disease forecasts. The model used here is supported by previous work (Cramer et al., 2021b), yet previous evaluation in a similar context has suggested that choice of baseline affects relative performance in general (Bracher et al., 2021b), and future research should be done on the best choices of baseline models in the context of infectious disease epidemics.

Our assessment of forecast performance may further have been inaccurate due to limitations in the observed data against which we evaluated forecasts. We sourced data from a globally aggregated database to maintain compatibility across 32 countries (Dong et al., 2020). However, this made it difficult to identify the origin of lags and inconsistencies between national data streams, and to what extent these could bias forecasts for different targets. In particular, we saw some real time data revised retrospectively, introducing bias in either direction where the data used to create forecasts was not the same as that used to evaluate it. We attempted to mitigate this by using an automated process for determining data revisions, and excluding forecasts made at a time of missing, unreliable, or heavily revised data. We also recognise that evaluating forecasts against updated data is a valid alternative approach used elsewhere (Cramer et al., 2021b). More generally it is unclear if the expectation of observation revisions should be a feature built into forecasts. Further research is needed to understand the perspective of end-users of forecasts in order to assess this.

The focus of this study was describing and summarising an ensemble of many models. We note that we have little insight into the individual methods and wide variety of assumptions that modellers used. While we asked modellers to provide a short description of their methods, we did not create a rigorous framework for this, and we did not document whether modellers changed the methods for a particular submitted model over time. Both the content of and variation in modelling methods and assumptions are likely to be critical to explaining performance, rather than describing or summarising it. Exploring modellers’ methods and relating this to forecast performance will be an important area of future work.

In an emergency setting, access to visualised forecasts and underlying data is useful for researchers, policymakers, and the public (CDC, 2020). Previous European multi-country efforts to forecast COVID-19 have included only single models adapted to country-specific parameters (Aguas et al., 2020; Adib et al., 2021; Agosto et al., 2021).

The European Forecasting Hub acted as a unique tool for creating an open-access, cross-country modelling network, and connecting this to public health policy across Europe. By opening participation to many modelling teams and with international high participation, we were able to create robust ensemble forecasts across Europe. This also allows comparison across forecasts built with different interpretations of current data, on a like for like scale in real time. The European Hub has supported policy outputs at an international, regional, and national level, including Hub forecasts cited weekly in ECDC Communicable Disease Threats Reports (European Centre for Disease Prevention and Control, 2022a).

For forecast producers, an easily accessible comparison between results from different methods can highlight individual strengths and weaknesses and help prioritise new areas of work. Collating time-stamped predictions ensures that we can test true out-of-sample performance of models and avoid retrospective claims of performance. Testing the limits of forecasting ability with these comparisons forms an important part of communicating any model-based prediction to decision makers. For example, the weekly ECDC Communicable Disease Threats reports include the specific results of this work by qualitatively highlighting the greater uncertainty around case forecasts compared to death forecasts.

This study raises many further questions which could inform epidemic forecast modellers and users. The dataset created by the European Forecast Hub is an openly accessible, standardised, and extensively documented catalogue of real time forecasting work from a range of teams and models across Europe (European Covid-19 Forecast Hub, 2023b), and we recommend its use for further research on forecast performance. In the code developed for this study, we provide a worked example of downloading and using both the forecasts and their evaluation scores (covid19-forecast-hub-europe, 2022).

Future work could explore the impact on forecast models of changing epidemiology at a broad spatial scale by combining analyses of trends and turning points in cases and deaths with forecast performance, or extending to include data on vaccination, variant, or policy changes over time. There is also much scope for future research into methods for combining forecasts to improve performance of an ensemble. This includes altering the inclusion criteria of forecast models based on different thresholds of past performance, excluding or including only forecasts that predict the lowest and highest values (trimming) (Taylor and Taylor, 2023), or using alternative weighting methods such as quantile regression averaging (Funk et al., 2020). Exploring these questions would add to our understanding of real time performance, supporting and improving future forecasting efforts.

We see additional scope to adapt the Hub format to the changing COVID-19 situation across Europe. We have extended the Forecast Hub infrastructure to include short term forecasts for hospitalisations with COVID-19, which is a challenging task due to limited data across the locations covered by the hub. As the policy focus shifts from immediate response to anticipating changes brought by vaccinations or the geographic spread of new variants (European Centre for Disease Prevention and Control, 2023), we are also separately investigating models for longer term scenarios in addition to the short term forecasts in a similar framework to existing scenario modelling work in the US (Borchering et al., 2021).

In conclusion, we have shown that during a rapidly evolving epidemic spreading through multiple populations, an ensemble forecast performed highly consistently across a large matrix of forecast targets, typically outperforming the majority of its separate component models and a naive baseline model. In addition, we have linked issues with the predictability of short-term case forecasts to underlying COVID-19 epidemiology, and shown that ensemble methods based on past model performance were unable to reliably improve forecast performance. Our work constitutes a step towards both unifying COVID-19 forecasts and improving our understanding of them.

Data availability

All source data were openly available before the study, originally available at: https://github.com/covid19-forecast-hub-europe/covid19-forecast-hub-europe (copy archived at swh:1:rev:b4d66c495e07c12d88384506154cf58f08592365). All data and code for this study are openly available on Github: covid19-forecast-hub-europe/euro-hub-ensemble.

References

  1. Preprint
    1. Cramer EY
    2. Ray EL
    3. Lopez VK
    4. Bracher J
    5. Brennen A
    6. Castro Rivadeneira AJ
    7. Gerding A
    8. Gneiting T
    9. House KH
    10. Huang Y
    11. Jayawardena D
    12. Kanji AH
    13. Khandelwal A
    14. Le K
    15. Mühlemann A
    16. Niemi J
    17. Shah A
    18. Stark A
    19. Wang Y
    20. Wattanachit N
    21. Zorn MW
    22. Gu Y
    23. Jain S
    24. Bannur N
    25. Deva A
    26. Kulkarni M
    27. Merugu S
    28. Raval A
    29. Shingi S
    30. Tiwari A
    31. White J
    32. Abernethy NF
    33. Woody S
    34. Dahan M
    35. Fox S
    36. Gaither K
    37. Lachmann M
    38. Meyers LA
    39. Scott JG
    40. Tec M
    41. Srivastava A
    42. George GE
    43. Cegan JC
    44. Dettwiller ID
    45. England WP
    46. Farthing MW
    47. Hunter RH
    48. Lafferty B
    49. Linkov I
    50. Mayo ML
    51. Parno MD
    52. Rowland MA
    53. Trump BD
    54. Zhang-James Y
    55. Chen S
    56. Faraone SV
    57. Hess J
    58. Morley CP
    59. Salekin A
    60. Wang D
    61. Corsetti SM
    62. Baer TM
    63. Eisenberg MC
    64. Falb K
    65. Huang Y
    66. Martin ET
    67. McCauley E
    68. Myers RL
    69. Schwarz T
    70. Sheldon D
    71. Gibson GC
    72. Yu R
    73. Gao L
    74. Ma Y
    75. Wu D
    76. Yan X
    77. Jin X
    78. Wang YX
    79. Chen Y
    80. Guo L
    81. Zhao Y
    82. Gu Q
    83. Chen J
    84. Wang L
    85. Xu P
    86. Zhang W
    87. Zou D
    88. Biegel H
    89. Lega J
    90. McConnell S
    91. Nagraj VP
    92. Guertin SL
    93. Hulme-Lowe C
    94. Turner SD
    95. Shi Y
    96. Ban X
    97. Walraven R
    98. Hong QJ
    99. Kong S
    100. van de Walle A
    101. Turtle JA
    102. Ben-Nun M
    103. Riley S
    104. Riley P
    105. Koyluoglu U
    106. DesRoches D
    107. Forli P
    108. Hamory B
    109. Kyriakides C
    110. Leis H
    111. Milliken J
    112. Moloney M
    113. Morgan J
    114. Nirgudkar N
    115. Ozcan G
    116. Piwonka N
    117. Ravi M
    118. Schrader C
    119. Shakhnovich E
    120. Siegel D
    121. Spatz R
    122. Stiefeling C
    123. Wilkinson B
    124. Wong A
    125. Cavany S
    126. España G
    127. Moore S
    128. Oidtman R
    129. Perkins A
    130. Kraus D
    131. Kraus A
    132. Gao Z
    133. Bian J
    134. Cao W
    135. Ferres JL
    136. Li C
    137. Liu TY
    138. Xie X
    139. Zhang S
    140. Zheng S
    141. Vespignani A
    142. Chinazzi M
    143. Davis JT
    144. Mu K
    145. y Piontti AP
    146. Xiong X
    147. Zheng A
    148. Baek J
    149. Farias V
    150. Georgescu A
    151. Levi R
    152. Sinha D
    153. Wilde J
    154. Perakis G
    155. Bennouna MA
    156. Nze-Ndong D
    157. Singhvi D
    158. Spantidakis I
    159. Thayaparan L
    160. Tsiourvas A
    161. Sarker A
    162. Jadbabaie A
    163. Shah D
    164. Penna ND
    165. Celi LA
    166. Sundar S
    167. Wolfinger R
    168. Osthus D
    169. Castro L
    170. Fairchild G
    171. Michaud I
    172. Karlen D
    173. Kinsey M
    174. Mullany LC
    175. Rainwater-Lovett K
    176. Shin L
    177. Tallaksen K
    178. Wilson S
    179. Lee EC
    180. Dent J
    181. Grantz KH
    182. Hill AL
    183. Kaminsky J
    184. Kaminsky K
    185. Keegan LT
    186. Lauer SA
    187. Lemaitre JC
    188. Lessler J
    189. Meredith HR
    190. Perez-Saez J
    191. Shah S
    192. Smith CP
    193. Truelove SA
    194. Wills J
    195. Marshall M
    196. Gardner L
    197. Nixon K
    198. Burant JC
    199. Wang L
    200. Gao L
    201. Gu Z
    202. Kim M
    203. Li X
    204. Wang G
    205. Wang Y
    206. Yu S
    207. Reiner RC
    208. Barber R
    209. Gakidou E
    210. Hay SI
    211. Lim S
    212. Murray CJL
    213. Pigott D
    214. Gurung HL
    215. Baccam P
    216. Stage SA
    217. Suchoski BT
    218. Prakash BA
    219. Adhikari B
    220. Cui J
    221. Rodríguez A
    222. Tabassum A
    223. Xie J
    224. Keskinocak P
    225. Asplund J
    226. Baxter A
    227. Oruc BE
    228. Serban N
    229. Arik SO
    230. Dusenberry M
    231. Epshteyn A
    232. Kanal E
    233. Le LT
    234. Li CL
    235. Pfister T
    236. Sava D
    237. Sinha R
    238. Tsai T
    239. Yoder N
    240. Yoon J
    241. Zhang L
    242. Abbott S
    243. Bosse NI
    244. Funk S
    245. Hellewell J
    246. Meakin SR
    247. Sherratt K
    248. Zhou M
    249. Kalantari R
    250. Yamana TK
    251. Pei S
    252. Shaman J
    253. Li ML
    254. Bertsimas D
    255. Lami OS
    256. Soni S
    257. Bouardi HT
    258. Ayer T
    259. Adee M
    260. Chhatwal J
    261. Dalgic OO
    262. Ladd MA
    263. Linas BP
    264. Mueller P
    265. Xiao J
    266. Wang Y
    267. Wang Q
    268. Xie S
    269. Zeng D
    270. Green A
    271. Bien J
    272. Brooks L
    273. Hu AJ
    274. Jahja M
    275. McDonald D
    276. Narasimhan B
    277. Politsch C
    278. Rajanala S
    279. Rumack A
    280. Simon N
    281. Tibshirani RJ
    282. Tibshirani R
    283. Ventura V
    284. Wasserman L
    285. O’Dea EB
    286. Drake JM
    287. Pagano R
    288. Tran QT
    289. Tung Ho LS
    290. Huynh H
    291. Walker JW
    292. Slayton RB
    293. Johansson MA
    294. Biggerstaff M
    295. Reich NG
    (2021b) Evaluation of Individual and Ensemble Probabilistic Forecasts of COVID-19 Mortality in the US
    medRxiv.
    https://doi.org/10.1101/2021.02.03.21250974

Article and author information

Author details

  1. Katharine Sherratt

    London School of Hygiene & Tropical Medicine, London, United Kingdom
    Contribution
    Conceptualization, Data curation, Software, Formal analysis, Investigation, Methodology, Writing - original draft, Writing – review and editing
    For correspondence
    katharine.sherratt@lshtm.ac.uk
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0003-2049-3423
  2. Hugo Gruson

    London School of Hygiene & Tropical Medicine, London, United Kingdom
    Contribution
    Software, Writing – review and editing
    Competing interests
    No competing interests declared
  3. Rok Grah

    European Centre for Disease Prevention and Control (ECDC), Stockholm, Sweden
    Contribution
    Funding acquisition, Project administration, Writing – review and editing
    Competing interests
    No competing interests declared
  4. Helen Johnson

    European Centre for Disease Prevention and Control (ECDC), Stockholm, Sweden
    Contribution
    Funding acquisition, Project administration, Writing – review and editing
    Competing interests
    No competing interests declared
  5. Rene Niehus

    European Centre for Disease Prevention and Control (ECDC), Stockholm, Sweden
    Contribution
    Funding acquisition, Project administration, Writing – review and editing
    Competing interests
    No competing interests declared
  6. Bastian Prasse

    European Centre for Disease Prevention and Control (ECDC), Stockholm, Sweden
    Contribution
    Funding acquisition, Project administration, Writing – review and editing
    Competing interests
    No competing interests declared
  7. Frank Sandmann

    European Centre for Disease Prevention and Control (ECDC), Stockholm, Sweden
    Contribution
    Funding acquisition, Project administration, Writing – review and editing
    Competing interests
    No competing interests declared
  8. Jannik Deuschel

    Karlsruhe Institute of Technology, Karlsruhe, Germany
    Contribution
    Software, Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  9. Daniel Wolffram

    Karlsruhe Institute of Technology, Karlsruhe, Germany
    Contribution
    Software, Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0003-0318-3669
  10. Sam Abbott

    London School of Hygiene & Tropical Medicine, London, United Kingdom
    Contribution
    Validation, Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  11. Alexander Ullrich

    Robert Koch Institute, Berlin, Germany
    Contribution
    Software, Visualization
    Competing interests
    No competing interests declared
  12. Graham Gibson

    University of Massachusetts Amherst, Amherst, United States
    Contribution
    Software, Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  13. Evan L Ray

    University of Massachusetts Amherst, Amherst, United States
    Contribution
    Software, Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  14. Nicholas G Reich

    University of Massachusetts Amherst, Amherst, United States
    Contribution
    Software, Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  15. Daniel Sheldon

    University of Massachusetts Amherst, Amherst, United States
    Contribution
    Software, Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  16. Yijin Wang

    University of Massachusetts Amherst, Amherst, United States
    Contribution
    Software, Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0003-4438-6366
  17. Nutcha Wattanachit

    University of Massachusetts Amherst, Amherst, United States
    Contribution
    Software, Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  18. Lijing Wang

    Boston Children’s Hospital and Harvard Medical School, Boston, United States
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  19. Jan Trnka

    Third Faculty of Medicine, Charles University, Prague, Czech Republic
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-1786-7562
  20. Guillaume Obozinski

    Ecole Polytechnique Federale de Lausanne, Lausanne, Switzerland
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  21. Tao Sun

    Ecole Polytechnique Federale de Lausanne, Lausanne, Switzerland
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0001-6357-6726
  22. Dorina Thanou

    Ecole Polytechnique Federale de Lausanne, Lausanne, Switzerland
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  23. Loic Pottier

    Éducation nationale, Valbonne, France
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  24. Ekaterina Krymova

    Eidgenossische Technische Hochschule, Zurich, Switzerland
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  25. Jan H Meinke

    Forschungszentrum Jülich GmbH, Jülich, Germany
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  26. Maria Vittoria Barbarossa

    Frankfurt Institute for Advanced Studies, Frankfurt, Germany
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  27. Neele Leithauser

    Fraunhofer Institute for Industrial Mathematics, Kaiserslautern, Germany
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  28. Jan Mohring

    Fraunhofer Institute for Industrial Mathematics, Kaiserslautern, Germany
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  29. Johanna Schneider

    Fraunhofer Institute for Industrial Mathematics, Kaiserslautern, Germany
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-9330-2838
  30. Jaroslaw Wlazlo

    Fraunhofer Institute for Industrial Mathematics, Kaiserslautern, Germany
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  31. Jan Fuhrmann

    Heidelberg University, Heidelberg, Germany
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-7091-3740
  32. Berit Lange

    Helmholtz Centre for Infection Research, Braunschweig, Germany
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  33. Isti Rodiah

    Helmholtz Centre for Infection Research, Braunschweig, Germany
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  34. Prasith Baccam

    IEM, Inc, Bel Air, United States
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    Affiliated with IEM, Inc. The author has no financial interests to declare
  35. Heidi Gurung

    IEM, Inc, Bel Air, United States
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    Affiliated with IEM, Inc. The author has no financial interests to declare
  36. Steven Stage

    IEM, Inc, Baton Rouge, United States
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    Affiliated with IEM, Inc. The author has no financial interests to declare
  37. Bradley Suchoski

    IEM, Inc, Bel Air, United States
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    Affiliated with IEM, Inc. The author has no financial interests to declare
  38. Jozef Budzinski

    Independent researcher, Vienna, Austria
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  39. Robert Walraven

    Independent researcher, Davis, United States
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  40. Inmaculada Villanueva

    Institut d’Investigacions Biomèdiques August Pi i Sunyer, Universitat Pompeu Fabra, Barcelona, Spain
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0003-4940-085X
  41. Vit Tucek

    Institute of Computer Science of the CAS, Prague, Czech Republic
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  42. Martin Smid

    Institute of Information Theory and Automation of the CAS, Prague, Czech Republic
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  43. Milan Zajicek

    Institute of Information Theory and Automation of the CAS, Prague, Czech Republic
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-3226-7266
  44. Cesar Perez Alvarez

    Inverence, Madrid, Spain
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  45. Borja Reina

    Inverence, Madrid, Spain
    Contribution
    Methodology, Writing – review and editing, Conceptualization
    Competing interests
    No competing interests declared
  46. Nikos I Bosse

    London School of Hygiene & Tropical Medicine, London, United Kingdom
    Contribution
    Software, Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  47. Sophie R Meakin

    London School of Hygiene & Tropical Medicine, London, United Kingdom
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  48. Lauren Castro

    Los Alamos National Laboratory, Los Alamos, United States
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  49. Geoffrey Fairchild

    Los Alamos National Laboratory, Los Alamos, United States
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  50. Isaac Michaud

    Los Alamos National Laboratory, Los Alamos, United States
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  51. Dave Osthus

    Los Alamos National Laboratory, Los Alamos, United States
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  52. Pierfrancesco Alaimo Di Loro

    LUMSA University, Rome, Italy
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  53. Antonello Maruotti

    LUMSA University, Rome, Italy
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0001-8377-9950
  54. Veronika Eclerova

    Masaryk University, Brno, Czech Republic
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0001-8476-7740
  55. Andrea Kraus

    Masaryk University, Brno, Czech Republic
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  56. David Kraus

    Masaryk University, Brno, Czech Republic
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  57. Lenka Pribylova

    Masaryk University, Brno, Czech Republic
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  58. Bertsimas Dimitris

    Massachusetts Institute of Technology, Cambridge, United States
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  59. Michael Lingzhi Li

    Massachusetts Institute of Technology, Cambridge, United States
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  60. Soni Saksham

    Massachusetts Institute of Technology, Cambridge, United States
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  61. Jonas Dehning

    Max-Planck-Institut für Dynamik und Selbstorganisation, Göttingen, Germany
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  62. Sebastian Mohr

    Max-Planck-Institut für Dynamik und Selbstorganisation, Göttingen, Germany
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  63. Viola Priesemann

    Max-Planck-Institut für Dynamik und Selbstorganisation, Göttingen, Germany
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0001-8905-5873
  64. Grzegorz Redlarski

    Medical University of Gdansk, Gdańsk, Poland
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  65. Benjamin Bejar

    Paul Scherrer Institute, Villigen, Switzerland
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  66. Giovanni Ardenghi

    Politecnico di Milano, Milan, Italy
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  67. Nicola Parolini

    Politecnico di Milano, Milan, Italy
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  68. Giovanni Ziarelli

    Politecnico di Milano, Milan, Italy
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  69. Wolfgang Bock

    Technical University of Kaiserlautern, Kaiserslautern, Germany
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  70. Stefan Heyder

    Technische Universität Ilmenau, Ilmenau, Germany
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  71. Thomas Hotz

    Technische Universität Ilmenau, Ilmenau, Germany
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  72. David E Singh

    Universidad Carlos III de Madrid, Leganes, Spain
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  73. Miguel Guzman-Merino

    Universidad Carlos III de Madrid, Leganes, Spain
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  74. Jose L Aznarte

    Universidad Nacional de Educación a Distancia (UNED), Madrid, Spain
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  75. David Morina

    Universitat de Barcelona, Barcelona, Spain
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  76. Sergio Alonso

    Universitat Politècnica de Catalunya, Barcelona, Spain
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-3989-8757
  77. Enric Alvarez

    Universitat Politècnica de Catalunya, Barcelona, Spain
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  78. Daniel Lopez

    Universitat Politècnica de Catalunya, Barcelona, Spain
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  79. Clara Prats

    Universitat Politècnica de Catalunya, Barcelona, Spain
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-1398-7559
  80. Jan Pablo Burgard

    Universitat Trier, Trier, Germany
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-5771-6179
  81. Arne Rodloff

    University of Cologne, Cologne, Germany
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  82. Tom Zimmermann

    University of Cologne, Cologne, Germany
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  83. Alexander Kuhlmann

    University of Halle, Halle, Germany
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  84. Janez Zibert

    University of Ljubljana, Ljubljana, Slovenia
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  85. Fulvia Pennoni

    University of Milano-Bicocca, Milano, Italy
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  86. Fabio Divino

    University of Molise, Pesche, Italy
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  87. Marti Catala

    University of Oxford, Oxford, United Kingdom
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  88. Gianfranco Lovison

    University of Palermo, Palermo, Italy
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  89. Paolo Giudici

    University of Pavia, Pavia, Italy
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  90. Barbara Tarantino

    University of Pavia, Pavia, Italy
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  91. Francesco Bartolucci

    University of Perugia, Perugia, Italy
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  92. Giovanna Jona Lasinio

    University of Rome "La Sapienza", Rome, Italy
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  93. Marco Mingione

    University of Rome "La Sapienza", Rome, Italy
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  94. Alessio Farcomeni

    University of Rome "Tor Vergata", Rome, Italy
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-7104-5826
  95. Ajitesh Srivastava

    University of Southern California, Los Angeles, United States
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  96. Pablo Montero-Manso

    University of Sydney, Sydney, Australia
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  97. Aniruddha Adiga

    University of Virginia, Charlottesville, United States
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  98. Benjamin Hurt

    University of Virginia, Charlottesville, United States
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  99. Bryan Lewis

    University of Virginia, Charlottesville, United States
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0003-0793-6082
  100. Madhav Marathe

    University of Virginia, Charlottesville, United States
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  101. Przemyslaw Porebski

    University of Virginia, Charlottesville, United States
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0001-8012-5791
  102. Srinivasan Venkatramanan

    University of Virginia, Charlottesville, United States
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  103. Rafal P Bartczuk

    University of Warsaw, Warsaw, Poland
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-0433-7327
  104. Filip Dreger

    University of Warsaw, Warsaw, Poland
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  105. Anna Gambin

    University of Warsaw, Warsaw, Poland
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  106. Krzysztof Gogolewski

    University of Warsaw, Warsaw, Poland
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0001-5523-5198
  107. Magdalena Gruziel-Slomka

    University of Warsaw, Warsaw, Poland
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  108. Bartosz Krupa

    University of Warsaw, Warsaw, Poland
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  109. Antoni Moszyński

    University of Warsaw, Warsaw, Poland
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  110. Karol Niedzielewski

    University of Warsaw, Warsaw, Poland
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  111. Jedrzej Nowosielski

    University of Warsaw, Warsaw, Poland
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  112. Maciej Radwan

    University of Warsaw, Warsaw, Poland
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  113. Franciszek Rakowski

    University of Warsaw, Warsaw, Poland
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  114. Marcin Semeniuk

    University of Warsaw, Warsaw, Poland
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  115. Ewa Szczurek

    University of Warsaw, Warsaw, Poland
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  116. Jakub Zielinski

    University of Warsaw, Warsaw, Poland
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0001-8935-8137
  117. Jan Kisielewski

    1. University of Warsaw, Warsaw, Poland
    2. University of Bialystok, Warsaw, Poland
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  118. Barbara Pabjan

    University of Wroclaw, Wroclaw, Poland
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  119. Kirsten Holger

    Universtät Leipzig, Leipzig, Germany
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  120. Yuri Kheifetz

    Universtät Leipzig, Leipzig, Germany
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  121. Markus Scholz

    Universtät Leipzig, Leipzig, Germany
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  122. Biecek Przemyslaw

    Warsaw University of Technology, Warsaw, Poland
    Present address
    University of Warsaw, Warsaw, Poland
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  123. Marcin Bodych

    Wroclaw University of Science and Technology, Wroclaw, Poland
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  124. Maciej Filinski

    Wroclaw University of Science and Technology, Wroclaw, Poland
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  125. Radoslaw Idzikowski

    Wroclaw University of Science and Technology, Wroclaw, Poland
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  126. Tyll Krueger

    Wroclaw University of Science and Technology, Wroclaw, Poland
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  127. Tomasz Ozanski

    Wroclaw University of Science and Technology, Wroclaw, Poland
    Contribution
    Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  128. Johannes Bracher

    Karlsruhe Institute of Technology, Karlsruhe, Germany
    Contribution
    Software, Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  129. Sebastian Funk

    London School of Hygiene & Tropical Medicine, London, United Kingdom
    Contribution
    Conceptualization, Software, Supervision, Writing – review and editing
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-2842-3406

Funding

Netzwerk Universitätsmedizin (Project egePan 01KX2021)

  • Jonas Dehning
  • Sebastian Mohr
  • Viola Priesemann

FISR (SMIGE - Modelli statistici inferenziali per governare l'epidemia, FISR 2020 - Covid-19 I Fase, FISR2020IP_00156, Codice Progetto - PRJ-0695)

  • Antonello Maruotti
  • Gianfranco Lovison
  • Alessio Farcomeni

Agència de Qualitat i Avaluació Sanitàries de Catalunya (Contract 2021_021OE)

  • Inmaculada Villanueva

European Centre for Disease Prevention and Control

  • Katharine Sherratt

European Commission (Communications Networks Content and Technology LC-01485746, Ministerio CIU/FEDER PGC2018-095456-B-I00)

  • Sergio Alonso
  • Enric Alvarez
  • Daniel Lopez
  • Clara Prats

Bundesministerium für Bildung und Forschung (05M18SIA)

  • Stefan Heyder
  • Thomas Hotz
  • Jan Pablo Burgard

Health Protection Research Unit (NIHR200908)

  • Nikos I Bosse

InPresa (Lombardy Region Italy)

  • Fulvia Pennoni
  • Francesco Bartolucci

Los Alamos National Laboratory

  • Lauren Castro

MUNI (Mathematical and Statistical modelling project (MUNI/A/1615/2020),MUNI/11/02202001/2020)

  • Veronika Eclerova
  • Lenka Pribylova

Ministerio de Sanidad

  • Cesar Perez Alvarez

Ministry of Science and Higher Education of Poland (28/WFSN/2021)

  • Rafal P Bartczuk

National Institute of General Medical Sciences (R35GM119582)

  • Graham Gibson

National Institutes of Health (1R01GM109718)

  • Lijing Wang

Virginia Department of Health (VDH-21-501-0141)

  • Aniruddha Adiga

Virginia Department of Health (VDH-21-501-0143)

  • Benjamin Hurt

Virginia Department of Health (VDH-21-501-0147)

  • Bryan Lewis

Virginia Department of Health (VDH-21-501-0142)

  • Lijing Wang

Virginia Department of Health (VDH-21-501-0148)

  • Madhav Marathe

Virginia Department of Health (VDH-21-501-0145)

  • Przemyslaw Porebski

Virginia Department of Health (VDH-21-501-0146)

  • Srinivasan Venkatramanan

Narodowe Centrum Badań i Rozwoju (INFOSTRATEG-I/0022/2021-00)

  • Biecek Przemyslaw

Horizon 2020 (PERISCOPE 101016233)

  • Paolo Giudici
  • Barbara Tarantino

German Free State of Saxony (LO-342/17-1)

  • Kirsten Holger
  • Yuri Kheifetz
  • Markus Scholz

Spanish Ministry of Health, Social Policy and Equality (REACT-UE (FEDER))

  • David E Singh

Wellcome Trust (210758/Z/18/Z)

  • Sam Abbott

RECETOX Přírodovědecké Fakulty Masarykovy Univerzity (LM2018121)

  • Veronika Eclerova

CETOCOEN EXCELLENCEC (CZ.02.1.01/0.0/0.0/17-043/0009632)

  • Veronika Eclerova

RECETOX RI project (CZ.02.1.01/0.0/0.0/16-013/0001761)

  • Veronika Eclerova

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication. For the purpose of Open Access, the authors have applied a CC BY public copyright license to any Author Accepted Manuscript version arising from this submission.

Version history

  1. Preprint posted: June 16, 2022 (view preprint)
  2. Received: July 18, 2022
  3. Accepted: February 20, 2023
  4. Accepted Manuscript published: April 21, 2023 (version 1)
  5. Version of Record published: June 2, 2023 (version 2)

Copyright

This is an open-access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication.

Metrics

  • 1,584
    Page views
  • 243
    Downloads
  • 10
    Citations

Article citation count generated by polling the highest count across the following sources: Crossref, PubMed Central, Scopus.

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Open citations (links to open the citations from this article in various online reference manager services)

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)

  1. Katharine Sherratt
  2. Hugo Gruson
  3. Rok Grah
  4. Helen Johnson
  5. Rene Niehus
  6. Bastian Prasse
  7. Frank Sandmann
  8. Jannik Deuschel
  9. Daniel Wolffram
  10. Sam Abbott
  11. Alexander Ullrich
  12. Graham Gibson
  13. Evan L Ray
  14. Nicholas G Reich
  15. Daniel Sheldon
  16. Yijin Wang
  17. Nutcha Wattanachit
  18. Lijing Wang
  19. Jan Trnka
  20. Guillaume Obozinski
  21. Tao Sun
  22. Dorina Thanou
  23. Loic Pottier
  24. Ekaterina Krymova
  25. Jan H Meinke
  26. Maria Vittoria Barbarossa
  27. Neele Leithauser
  28. Jan Mohring
  29. Johanna Schneider
  30. Jaroslaw Wlazlo
  31. Jan Fuhrmann
  32. Berit Lange
  33. Isti Rodiah
  34. Prasith Baccam
  35. Heidi Gurung
  36. Steven Stage
  37. Bradley Suchoski
  38. Jozef Budzinski
  39. Robert Walraven
  40. Inmaculada Villanueva
  41. Vit Tucek
  42. Martin Smid
  43. Milan Zajicek
  44. Cesar Perez Alvarez
  45. Borja Reina
  46. Nikos I Bosse
  47. Sophie R Meakin
  48. Lauren Castro
  49. Geoffrey Fairchild
  50. Isaac Michaud
  51. Dave Osthus
  52. Pierfrancesco Alaimo Di Loro
  53. Antonello Maruotti
  54. Veronika Eclerova
  55. Andrea Kraus
  56. David Kraus
  57. Lenka Pribylova
  58. Bertsimas Dimitris
  59. Michael Lingzhi Li
  60. Soni Saksham
  61. Jonas Dehning
  62. Sebastian Mohr
  63. Viola Priesemann
  64. Grzegorz Redlarski
  65. Benjamin Bejar
  66. Giovanni Ardenghi
  67. Nicola Parolini
  68. Giovanni Ziarelli
  69. Wolfgang Bock
  70. Stefan Heyder
  71. Thomas Hotz
  72. David E Singh
  73. Miguel Guzman-Merino
  74. Jose L Aznarte
  75. David Morina
  76. Sergio Alonso
  77. Enric Alvarez
  78. Daniel Lopez
  79. Clara Prats
  80. Jan Pablo Burgard
  81. Arne Rodloff
  82. Tom Zimmermann
  83. Alexander Kuhlmann
  84. Janez Zibert
  85. Fulvia Pennoni
  86. Fabio Divino
  87. Marti Catala
  88. Gianfranco Lovison
  89. Paolo Giudici
  90. Barbara Tarantino
  91. Francesco Bartolucci
  92. Giovanna Jona Lasinio
  93. Marco Mingione
  94. Alessio Farcomeni
  95. Ajitesh Srivastava
  96. Pablo Montero-Manso
  97. Aniruddha Adiga
  98. Benjamin Hurt
  99. Bryan Lewis
  100. Madhav Marathe
  101. Przemyslaw Porebski
  102. Srinivasan Venkatramanan
  103. Rafal P Bartczuk
  104. Filip Dreger
  105. Anna Gambin
  106. Krzysztof Gogolewski
  107. Magdalena Gruziel-Slomka
  108. Bartosz Krupa
  109. Antoni Moszyński
  110. Karol Niedzielewski
  111. Jedrzej Nowosielski
  112. Maciej Radwan
  113. Franciszek Rakowski
  114. Marcin Semeniuk
  115. Ewa Szczurek
  116. Jakub Zielinski
  117. Jan Kisielewski
  118. Barbara Pabjan
  119. Kirsten Holger
  120. Yuri Kheifetz
  121. Markus Scholz
  122. Biecek Przemyslaw
  123. Marcin Bodych
  124. Maciej Filinski
  125. Radoslaw Idzikowski
  126. Tyll Krueger
  127. Tomasz Ozanski
  128. Johannes Bracher
  129. Sebastian Funk
(2023)
Predictive performance of multi-model ensemble forecasts of COVID-19 across European nations
eLife 12:e81916.
https://doi.org/10.7554/eLife.81916

Share this article

https://doi.org/10.7554/eLife.81916

Further reading

    1. Ecology
    2. Epidemiology and Global Health
    Aleksandra Kovacevic, David RM Smith ... Lulla Opatowski
    Research Article

    Non-pharmaceutical interventions implemented to block SARS-CoV-2 transmission in early 2020 led to global reductions in the incidence of invasive pneumococcal disease (IPD). By contrast, most European countries reported an increase in antibiotic resistance among invasive Streptococcus pneumoniae isolates from 2019 to 2020, while an increasing number of studies reported stable pneumococcal carriage prevalence over the same period. To disentangle the impacts of the COVID-19 pandemic on pneumococcal epidemiology in the community setting, we propose a mathematical model formalizing simultaneous transmission of SARS-CoV-2 and antibiotic-sensitive and -resistant strains of S. pneumoniae. To test hypotheses underlying these trends five mechanisms were built into the model and examined: (1) a population-wide reduction of antibiotic prescriptions in the community, (2) lockdown effect on pneumococcal transmission, (3) a reduced risk of developing an IPD due to the absence of common respiratory viruses, (4) community azithromycin use in COVID-19 infected individuals, (5) and a longer carriage duration of antibiotic-resistant pneumococcal strains. Among 31 possible pandemic scenarios involving mechanisms individually or in combination, model simulations surprisingly identified only two scenarios that reproduced the reported trends in the general population. They included factors (1), (3), and (4). These scenarios replicated a nearly 50% reduction in annual IPD, and an increase in antibiotic resistance from 20% to 22%, all while maintaining a relatively stable pneumococcal carriage. Exploring further, higher SARS-CoV-2 R0 values and synergistic within-host virus-bacteria interaction mechanisms could have additionally contributed to the observed antibiotic resistance increase. Our work demonstrates the utility of the mathematical modeling approach in unraveling the complex effects of the COVID-19 pandemic responses on AMR dynamics.

    1. Epidemiology and Global Health
    Olivera Djuric, Elisabetta Larosa ... The Reggio Emilia Covid-19 Working Group
    Research Article

    Background:

    The aim of our study was to test the hypothesis that the community contact tracing strategy of testing contacts in households immediately instead of at the end of quarantine had an impact on the transmission of SARS-CoV-2 in schools in Reggio Emilia Province.

    Methods:

    We analysed surveillance data on notification of COVID-19 cases in schools between 1 September 2020 and 4 April 2021. We have applied a mediation analysis that allows for interaction between the intervention (before/after period) and the mediator.

    Results:

    Median tracing delay decreased from 7 to 3.1 days and the percentage of the known infection source increased from 34–54.8% (incident rate ratio-IRR 1.61 1.40–1.86). Implementation of prompt contact tracing was associated with a 10% decrease in the number of secondary cases (excess relative risk –0.1 95% CI –0.35–0.15). Knowing the source of infection of the index case led to a decrease in secondary transmission (IRR 0.75 95% CI 0.63–0.91) while the decrease in tracing delay was associated with decreased risk of secondary cases (1/IRR 0.97 95% CI 0.94–1.01 per one day of delay). The direct effect of the intervention accounted for the 29% decrease in the number of secondary cases (excess relative risk –0.29 95%–0.61 to 0.03).

    Conclusions:

    Prompt contact testing in the community reduces the time of contact tracing and increases the ability to identify the source of infection in school outbreaks. Although there are strong reasons for thinking it is a causal link, observed differences can be also due to differences in the force of infection and to other control measures put in place.

    Funding:

    This project was carried out with the technical and financial support of the Italian Ministry of Health – CCM 2020 and Ricerca Corrente Annual Program 2023.