Forecasting the spatial spread of an Ebola epidemic in real time: Comparing predictions of mathematical models and experts
- Centre for Mathematical Modelling of Infectious Diseases, London School of Hygiene and Tropical Medicine, United Kingdom
- Department of Infectious Disease Epidemiology, London School of Hygiene and Tropical Medicine, United Kingdom
- Department of Biosystems Science and Engineering, ETH Zürich, Switzerland
Figures
Figure 1
The extent of the 2018–2020 Ebola outbreak in the northeastern Democratic Republic of the Congo (DRC) and areas included in our study.
(A) Daily incidence in the northeastern DRC between August 2018 and March 2020. Grey points show days prior to the study period, coloured points show days within the study period (November 2019–March 2020), and hue indicates month. (B) shows the total number of cases of Ebola recorded in each health zone. (C) Number of cases in each month and health zone during the period covered by this study; health zones outlined in red show all health zones affected by the entire epidemic.
Figure 2
Health zones included in the model and expert elicitation survey.
(A) shows the provinces around the affected area, and included in the transmission model, and the red box shows the area detailed in panel (B). (B) shows the health zones included at least once in the expert elicitation survey we conducted.
Figure 3 with 1 supplement
Timeline of the expert elicitation.
Each point shows the date of the interview of the expert labelled to obtain forecasts for the following month. Colour indicates the month for which the forecast was made, and the forecast windows are highlighted with a shaded band of the same colour.
Figure 3—figure supplement 1
Tools used for expert elicitation.
On the left, a screenshot of the interactive map provided to experts. On the right, a screenshot of the MATCH expert elicitation tool.
Figure 4 with 4 supplements
Expert elicitation results and accuracy of predictions.
Only the health zones (HZs) that were rated by all experts are included here. Results are shown as probabilities (vertical axes) that a given HZ (horizontal panels) exceeds a given threshold (horizontal axes) according to the experts (box plots) or models (square/diamond for gravity and adjacency models, respectively) across different months (vertical panels). HZ/month combinations where the given thresholds were exceeded are marked in cyan, and ones where they were not are marked in red.
Figure 4—figure supplement 1
Expert forecasts made using MATCH for the distribution of cases expected in the month of December.
Figure 4—figure supplement 2
Expert forecasts made using MATCH for the distribution of cases expected in the month of January.
Figure 4—figure supplement 3
Expert forecasts made using MATCH for the distribution of cases expected in the month of February.
Figure 4—figure supplement 4
Expert forecasts made using MATCH for the distribution of cases expected in the month of March.
Figure 5 with 1 supplement
Evaluation of forecasts made by the experts, models, and ensembles.
(A) shows the overall Brier score for each expert, model, and ensemble (calculated over all forecasts included in the study). In (B), each panel shows the Brier score across all health zones for each month (vertical) at each case threshold (horizontal). Coloured points show each expert score, and the violin plot shows their distribution. The grey hollow points show the model scores, the yellow points show the ensemble scores (circles show experts alone, squares show models alone, and triangles show experts and models with 50% weight given to each). (C) shows the ranking of each expert and model in terms of forecast performance.
Figure 5—figure supplement 1
Risk rank of health zones.
Each panel shows the rank of the health zones based on expert ensemble, expert, and model ensemble and model forecast probabilities of exceeding each case threshold (horizontal panels) in each month (vertical panels). Coloured lines and points indicate the health zone rank. Solid lines show health zones where the case threshold was met, and faded lines show those where the case threshold was not met.
Figure 6
Bias and calibration of forecasts.
Panels show the hazards gap difference between the hazard rate (expected number of exceedances across all health zones) for each threshold attributed by the forecast and the actual number of health zones that exceeded the associated threshold. Each panel shows one forecast (expert or model) in each month. The bottom row shows the same for each forecast calculated over the entire study period.
Figure 7
Evaluation of forecasts made in health zones not included in the main survey.
Each panel (right to left) shows the Brier score across all health zones for each month. Coloured points show each expert score, and the density plot shows their overall distribution. The red points show the model scores, and the yellow points show the ensemble scores (squares show experts alone, and crosses show experts and models with 50% weight given to each).
Tables
Table 1
Experts and health zones included in each round of the survey.
The left part of the table details the experts interviewed (highlighted in green) the health zones included in the main survey in each month. In addition, the right part of the table details the health zones nominated by experts and the number of experts that nominated each one.
| Month forecasted | Experts interviewed (highlighted) | Health zones in interview (HZs) | No. of experts | HZ nominated | ||||
|---|---|---|---|---|---|---|---|---|
| December | 1 | 2 | 3 | Beni | 7 | Oicha | ||
| 4 | 5 | 6 | Goma | 5 | Komanda | |||
| 7 | 8 | 9 | Kalunguta | 2 | Butembo | |||
| 10 | 11 | 12 | Mabalako | 2 | Katwa | |||
| 13 | 14 | Mambasa | 1 | Lolwa | ||||
| Mandima | 1 | Makiso-Kisangani | ||||||
| 1 | Nyankunde | |||||||
| January | 1 | 2 | 3 | Beni | 6 | Butembo | ||
| 4 | 5 | 6 | Biena | 3 | Katwa | |||
| 7 | 8 | 9 | Goma | 1 | Kalunguta | |||
| 10 | 11 | 12 | Mabalako | 1 | Mangurerdjipa | |||
| 13 | 14 | Mandima | ||||||
| Oicha | ||||||||
| February | 1 | 2 | 3 | Beni | 6 | Oicha | ||
| 4 | 5 | 6 | Bunia | 4 | Biena | |||
| 7 | 8 | 9 | Butembo | 2 | Vuhovi | |||
| 10 | 11 | 12 | Goma | 1 | Lolwa | |||
| 13 | 14 | Kalunguta | ||||||
| Katwa | ||||||||
| Kayna | ||||||||
| Mabalako | ||||||||
| Mambasa | ||||||||
| Mandima | ||||||||
| Musienene | ||||||||
| March | 1 | 2 | 3 | Beni | 3 | Mabalako | ||
| 4 | 5 | 6 | Butembo | |||||
| 7 | 8 | 9 | Goma | |||||
| 10 | 11 | 12 | Mandima | |||||
| 13 | 14 | |||||||
Additional files
Download links
A two-part list of links to download the article, or parts of the article, in various formats.
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
Forecasting the spatial spread of an Ebola epidemic in real time: Comparing predictions of mathematical models and experts
eLife 13:RP98005.
https://doi.org/10.7554/eLife.98005.3
