The extent of the 2018-2020 ebola outbreak in north-eastern DRC and areas included in our study.

A) Daily incidence in north-eastern DRC between August 2018 and March 2020. Grey points show days prior to the study period, coloured points show days within the study period (November 2019 - March 2020), hue indicates month. B) Shows the total number of cases of ebola recorded in each Health Zone. C) Number of cases in each month and health zone during the period covered by this study, health zones outlined in red show all health zones affected by the entire epidemic.

Health zones included in the model and Expert Elicitation survey.

A shows the provinces around the affected area, and included in the transmission model, the red box shows the area detailed in panel B. B shows the health zones included at least once in the Expert Elicitation survey we conducted.

Timeline of the expert elicitation.

Each point shows the date of the interview of the expert labelled to obtain forecasts for the following month. Colour indicates the month for which the forecast was made, the forecast windows are highlighted with a shaded band of the same colour.

Expert elicitation results and accuracy of predictions.

Only the HZs that were rated by all experts are included here. Results are shown as probabilities (vertical axes) that a given health zone (horizontal panels) exceeds a given threshold (horizontal axes) according to the experts (box plots) or models (square / diamond for gravity and adjacency models, respectively) across different months (vertical panels). Health zone / month combinations where the given thresholds were exceeded are marked in cyan, and ones where they weren’t in red.

Evaluation of forecasts made by the experts, models, and ensembles.

A shows the overall Brier Score for each expert, model and ensemble (calculated over all forecasts included in the study). In B each panel shows the Brier score across all health zones for each month (vertical) at each case threshold (Horizontal). Coloured points show each expert score, the violin plot shows their distribution. The grey hollow points show the model scores, the yellow points show the ensemble scores (circles show experts alone, squares show models alone and triangles show experts and models with 50% weight given to each). C Shows the ranking of each expert and model in terms of forecast performance

Bias and calibration of forecasts.

Panels show the Hazards gap difference between the Hazard rate (expected number of exceedances across all health zones) for each threshold attributed by the forecast and the actual number of Health Zones that exceeded the associated threshold. Each panel shows one forecast (expert or model) in each month. The bottom row shows the same for each forecast calculated over the entire study period.

Experts and health zones included in each round of the survey

Evaluation of forecasts made in health zones not included in the main survey.

Each panel (right to left) shows the Brier score across all health zones for each month. Coloured points show each expert score, density plot shows their overall distribution. The red points show the model scores, the yellow points show the ensemble scores (squares show experts alone, crosses show experts and models with 50% weight given to each).