The rapid development, scale-up, and deployment of COVID-19 vaccines in the United States (US) has been one of the biggest public health successes in the US during this pandemic, with reported cases in a nadir in June 2021 (Centers for Disease Control and Prevention, 2021b), despite increased testing capacities. With this success, non-pharmaceutical interventions (NPIs) were lifted, including mask mandates, in almost every jurisdiction across the US in Spring 2021. However, the emergence of novel variants with increased transmissibility and /or immune escape, particularly the Delta and Omicron variants, has continually raised concern about the potential timing and magnitude of the subsequent resurgence, and the ability to mitigate it through increased uptake of vaccination.

Established in December 2020, the COVID-19 Scenario Modeling Hub is an effort to apply a multiple-model approach to produce six-month projections of the state and national trajectories of cases, hospitalizations, and deaths in the US under defined scenarios (Borchering et al., 2021). Scenarios from projection rounds have focused on control measures, vaccination availability and uptake, emerging variants, and waning immunity (COVID-19 Scenario Modeling Hub, 2020). Projections are released in a timely manner to guide policy decisions and data are made publicly available on a website (COVID-19 Scenario Modeling Hub, 2020).

Here, we detail results from the seventh round of projections, in which increased transmissibility variants were incorporated into projections to assess the potential impact of the Delta variant. In all scenarios, resurgences across the US were projected, with the largest resurgences occurring for scenarios with the highest variant transmissibility (60% increase over the Alpha variant, which most closely resembles estimates for the Delta variant). Corresponding increases in hospitalizations and deaths were also projected. In scenarios with higher vaccine coverage, the size and duration of this resurgence was notably smaller. Cases were projected to increase in early July 2021 at the national level and peak in mid to late September 2021. Corresponding increases in hospitalizations and deaths were also projected. The resurgence was projected to be geographically heterogeneous; although most states were projected to experience some degree of rebound, those having higher vaccine coverage were projected to experience less severe increases in incidence relative to prior observed peaks. However, while the timing of these projected resurges was relatively accurate compared to what has since been reported, the magnitude of reported cases, hospitalization, and deaths far surpassed what was projected. Here, we describe our experience with multi-model projections of the Delta variant to highlight both the value of scenario-based projections for planning, but also the challenges to understand and predict the constantly evolving COVID-19 pandemic.

In the two scenarios with high Delta variant transmissibility (60% more transmissible than Alpha), we projected a national wave of cases to continue to grow over the summer and peak in mid- to late September 2021. In the scenario that assumes lower vaccination coverage among eligible individuals (70%) and higher variant transmissibility (the most pessimistic scenario), this resurgence was projected to peak at 414,000 weekly cases (95% projection interval (PI): 140,000–1,525,000) and 5900 weekly deaths (95% PI: 900–30,000) nationally. Overall, this scenario projected 7,554,000 (95% PI: 3,294,000–28,399,000) cumulative cases and 96,000 (95% PI: 27,000–476,000) cumulative deaths during July 4, 2021–Jan 1, 2022 (Figure 1).

Figure 1 with 3 supplements see all

Download asset Open asset

With higher variant transmissibility, increasing national vaccination coverage was projected to temper the fall wave slightly and cause it to drop more quickly, but not prevent it. With an increase in national vaccine coverage to 80% by January 1, 2022, the ensemble projected 65,000 (16%) fewer cases and 1300 (21%) fewer deaths per week at the peak, and 1,525,000 (20%) and 21,000 (22%) fewer cumulative cases and deaths, respectively, during July 4, 2021–January 1, 2022, when compared to the scenario where vaccination saturated at 70% nationally (Figure 1).

The projected national resurgence in COVID-19 cases in the higher transmissibility variant scenarios was composed of highly heterogeneous state-level resurgences. The ten states with the largest projected increases in incidence relative to their winter 2020–21 peak were, in descending order, Louisiana, Hawaii, Nevada, Arkansas, Florida, Missouri, Georgia, Alabama, Alaska, and Arizona. The ensemble estimates projected these states to experience median peak levels of weekly incident cases that were 18–69% (95% PI: 1%–541%) of their (smoothed) winter peak, although this was exceeded in many states. The 10 states with the lowest projected resurgences were, in ascending order, Massachusetts, Rhode Island, Vermont, Maine, Minnesota, Pennsylvania, North Dakota, Wisconsin, Tennessee, and South Dakota. The 10 states with the smallest projected resurgence had a median first-dose vaccine coverage of 70% among the eligible population (ages 12+) on July 3, 2021, compared to 56% in the ten states with the highest projected resurgence. We find a high negative correlation (Pearson’s r=–0.66, Figure 2) between projected cumulative deaths per population and vaccination coverage on July 3, 2021 (NCIRD, 2021). In all states, even those with low overall vaccination coverage, at least 76% of people 65+had received at least one dose of the vaccine, which was expected to have a major effect in limiting mortality from Delta (Centers for Disease Control and Prevention, 2021b).

Figure 2

Download asset Open asset

The impact of vaccination was already being observed early in the Delta wave: in the 10 states with the largest projected resurgence there was a 9% reduction in the observed case fatality ratio (CFR) comparing August–December 2020 and January–July 2021; in the 10 states with the smallest projected resurgence a 21% reduction in CFR was observed. During the projection period, we projected CFR reductions of 15% and 14%, as compared to August-December 2020. Lower transmissibility variant scenarios projected significantly reduced resurgence, projecting cumulative national cases of only 9% and 13% compared to the winter 2020–21 peak. Similarly, in the previous projection round (Round 6), which similar to the 7th round except it assumed only a 20% transmissibility increase from a novel variant, resurgence was expected to produce only 8% of the cases reported during the winter 2020–21 peak nationally (COVID-19 Scenario Modeling Hub, 2020).

Weekly case observations exceeded our 95% projection interval in the first 9 weeks of the projection period in all scenarios, including those assuming 60% increased transmissibility of the Delta variant (Figure 1). Our projections also tended to underestimate hospitalizations and deaths in the ascending phase of the Delta wave, although to a lesser degree. We compared weekly incident and cumulative cases during the first four weeks after the projection date (July 4–31, 2021). The total median projected number of cases underestimated the observed cases overall during this 4-week period (1,256,000 observed vs 516,000 projected); however, we find a strong correlation between ranking of observed and projected total cases per 100,000 during the first four weeks of the projection period, at the state level (Spearman’s ⍴=0.87, Figure 3). Seven of the ten states with greatest projected incidence rank in the ten worst observed incidence states. Hence while projections did not capture the full scope of the rise in incidence due to the Delta variant, these projections reflected the expected severity ranking among state projections well.

Figure 3

Download asset Open asset

The two high variant transmissibility scenarios, which most closely resembled the characteristics of the circulating Delta variant, projected the timing of the Delta resurgence, projecting deaths to increase simultaneously with reported cases and peak one week after reported cases (Figure 1). However, all scenarios substantially underestimated the magnitude of the Delta wave for all outcomes. Among the two high variant transmissibility scenarios, at the national level the peak cases were under-projected by 70% (95% PI: −25–91%) and 64% (95% PI: −49–88%) (349,183 and 413,733 vs 1.15 M peak reported weekly cases), though the 95% projection interval did capture the reported magnitude of the peak (Figure 1). Similarly, hospitalizations and deaths were also under-projected by 47% (95% PI: −184–84%) and 36% (95% PI: −230–82%) (46,000 and 56,000 vs 87,000 peak reported hospitalizations) and 67% (95% PI: −77–93%) and 59% (95% PI: −120–93%) (4700 and 6000 vs 14,000 peak reported deaths), respectively; both also captured the reported magnitude within projection intervals.

Prevalence of the SARS-CoV-2 Delta variant rose quickly in the US between May and June 2021, with the variant achieving dominance by late June 2021, and accounting for over 90% of all SARS-CoV-2 infections for an extended period from late July to early December 2021 (Centers for Disease Control and Prevention, 2021b). This variant prompted concerns about the scale of the COVID-19 resurgence in the US in the summer and fall of 2021, especially in the midst of decreased NPIs and slowing vaccination rates. Projections combining insights from multiple models suggested sizable resurgences of COVID-19 across the US, assuming growth of a variant that is 60% more transmissible than the Alpha variant (an assumption aligned with most estimates of the relative transmissibility of the Delta variant) (Allen et al., 2022). In scenarios with higher vaccination coverage, the magnitude of the resurgence in cases and deaths was substantially lower than in the lower coverage scenarios. Efforts to increase vaccination rates are critical and will save lives before and during future resurgences. At the outset of the Delta wave on July 1, 2021, only 13 states and Washington, D.C. had accomplished President Biden’s goal for vaccination coverage among eligible populations at or above 70%.

The rapid case growth observed in July 2021 in multiple US states was surprising, tracking with or above the projections from our worst-case scenario. Scenarios were designed at the end of June 2021 based on information available at that time about the transmissibility of the Delta variant, vaccine effectiveness, and vaccine coverage; projections were generated on or before July 4, 2021. Our data should not be understood as forecasts, but as projections conditional on the scenarios and assumptions; several reasons could explain why case growth was faster than expected, including key epidemiological and behavioral aspects that may have affected the disease dynamics. Possible mechanisms driving the underestimation of the observed Summer 2021 resurgence may include inaccurate assumptions about the transmissibility and severity of Delta (including changes in serial interval or severity of infection relative to other variants), the effectiveness of the vaccines against infection and transmission, waning of natural and vaccine-derived immunity, changes in testing practices, and the interaction of these factors with NPIs and behavior change (Li et al., 2021; Elliott et al., 2021; Puranik et al., 2021). Assumptions regarding these factors were left to the discretion of the teams and therefore vary across models. Critically, eight of the nine teams did not include waning of natural or vaccine-derived immunity, an assumption that we now know is incorrect. (Higdon et al., 2021) Subsequent rounds of projection have focused on different scenarios of waning immunity. Yet in this round, absence of waning resulted in a much smaller population susceptible to infection with Delta, thus reducing the overall potential magnitude that could be projected by these models. Use of NPIs has been substantially reduced across the US, with a lapse in mask mandates in most states. Although modeled use of NPIs was left to the discretion of individual modeling teams and did not vary between scenarios, results of prior rounds underscore the effectiveness of NPIs, in combination with increasing vaccination, to moderate the spread of a highly transmissible variant (Borchering et al., 2021). The extent to which NPIs may be necessary will vary across states, as those states with high levels of vaccination coverage or natural immunity may be at lower risk for an increase in cases.

The impact of the resurgence on severe disease and mortality was expected to vary substantially across states; states with younger populations and higher vaccination coverage among older and high-risk populations were expected to experience a relatively lower burden of severe disease, even with resurgences in cases. Several states (for example, South Dakota, North Dakota) with low vaccination coverage were not projected to experience major resurgences, likely because of high naturally acquired immunity. In addition, as the projected resurgence continued throughout the summer and into the beginning of the school year, efforts to promote vaccination among eligible school-aged children and college students and to maintain key prevention strategies in schools (for example, mask-wearing among the unvaccinated, physical distancing, screening programs) likely helped reduce risks with a safe return to in-person instruction (Centers for Disease Control and Prevention, 2021a). Observed increases in new vaccinations, particularly among young age groups and in jurisdictions most severely impacted by the Delta variant, was a positive step in this direction (Centers for Disease Control and Prevention, 2021b).

The findings in this report are subject to several limitations. First, considerable uncertainty is inherent to long-term projections. This has been repeatedly illustrated throughout the COVID-19 pandemic, with rapid changes in behavior, deployment of vaccines and boosters, and the emergence of novel variants, each of which has the capacity to drastically shift the epidemic trajectories. Uncertainty may arise from three main sources: specification of the scenarios (for example, uncertainty in transmissibility); errors in the structure or assumptions of individual models given a specific scenario (for example, variations in assumptions about vaccination uptake); and inaccurate calibration based on incomplete or biased data (for example, reporting backlogs). None of the four scenarios considered here were likely to precisely reflect the future reality over a 6-month period. As a case in point the emergence of the Omicron variant in December 2021, at the end of our projection period, could not have been predicted when scenarios were designed in June 2021 (Borchering et al., 2021). Similarly, a resurgence in Delta variant incidence was observed in mid-fall 2021, possibly due to changes in behavior and waning immunity, and is not captured in scenarios or model projections. Further, for a given scenario, there is notable variation among individual model projections with regard to both the timing and the magnitude of the resurgence (Figure 1—figure supplements 1–3). Variation likely reflects differences in model structure, projected vaccine coverage, projected variant growth, and importance of seasonal effects. Some of these variations reflect true scientific uncertainty, making ensemble projections particularly useful to integrate uncertainty between and within individual models. In addition, these scenarios do not specify considerations of Delta infecting previously immune individuals due to moderate antigenic changes, the waning of existing immunity, increases in NPIs, or vaccination among children aged <12 years starting in November 2021, all of which were expected to be important drivers of dynamics in the subsequent months. In the same vein, model estimates are dependent on assumptions about vaccine hesitancy, which are informed in part by large-scale surveys of vaccine sentiments (Carnegie Mellon University Delphi Group, 2021; Estimates of Vaccine Hesitancy for COVID-19, 2021). These surveys may underestimate vaccine hesitancy, as coverage estimates among survey respondents are substantially higher than measured among the overall US population. Additionally, there are limitations to individual component models, although these concerns are tempered by analyzing ensembles of the nine different models. Overall, a full evaluation of our projections and sources of uncertainty is particularly difficult in a scenario context and is beyond the scope of this paper. However, it is worth noting that in this particular round of projection, the relationship between projection accuracy and time horizon is not straightforward (e.g. refer to Figure 1 for a visual assessment of coverage).

The COVID-19 Scenario Modeling Hub (COVID-19 Scenario Modeling Hub, 2020) convened nine modeling teams in an open call to provide six-month (July 3, 2021-January 1, 2022) COVID-19 projections in the US using data available through July 3, 2021. Each team developed a model to project weekly reported cases, hospitalizations, and deaths, both nationally and by jurisdiction (50 states and the District of Columbia), for four different epidemiological scenarios. Models were calibrated against data from the Johns Hopkins Center for Systems Science and Engineering Coronavirus Resource Center and federal databases (Coronavirus Resource Center, 2020; US Department of Health and Human Services, 2020). The four scenarios included low and high vaccination hesitancy levels, assuming national vaccination coverage saturation at 80% and 70%, respectively, based on hesitancy surveys (Table 1) (Carnegie Mellon University Delphi Group, 2021; Estimates of Vaccine Hesitancy for COVID-19, 2021). Participating teams accounted for vaccination rates by state, age, and risk-groups (for example, older adults and health care workers). Specified vaccine efficacy levels were constant across the scenarios and were based on protection against clinical disease in randomized clinical trials and effectiveness studies; parameters for effectiveness against infection, transmission, and progression to severe outcomes (for example, death) were left to be specified by each team (COVID-19 Scenario Modeling Hub, 2020). When the scenarios were designed in late June 2021, little information was available on vaccine efficacy specific to the Delta variant and on waning immunity. For details on individual model assumptions, see Supplementary file 1.

Scenarios assumed one of two levels of increased transmissibility for the Delta variant: 40% (low) or 60% (high) more transmissible than the Alpha variant. Increases in new variant prevalence over time were determined by each modeling team and were estimated at the state level.

Individual models differed substantially in structure and design; see Supplementary file 1 and the COVID-19 Scenario Modeling Hub GitHub website for more details (COVID-19 Scenario Modeling Hub, 2021). Individual modeling teams provided probabilistic projections of incident and cumulative epidemic trajectory for each week of the projection period, with 23 quantiles requested (0.01, 0.025, 0.05, every 5%–0.95, 0.975, and 0.99). These individual projections were combined into an ensemble for each scenario, outcome, week, and location using an equally-weighted linear opinion pool method across teams that trimmed the highest and lowest model at each point and quantile (Stone, 1961; Jose et al., 2014). Point estimates provided here are the median of the ensemble.

For any given pair of scenarios, averted cases and deaths were calculated as the difference (and ratio) between the median point estimates of the ensemble for the two scenarios. To provide a relative measure of resurgence in each state, we compared the intensity of the projected outbreak in the next six months to the size of the winter 2020–2021 outbreak – a period of high hospital burden in many jurisdictions. Specifically, projected resurgences were assessed by taking the ratio of the peak projected median incidence in a given location over the projection period (July 3, 2021-January 1, 2022) to the highest incidence experienced during the winter 2020–2021 period (defined as October 1, 2020–February 28, 2021) for the same location. Winter 2020–21 peaks were identified as the seven-day average centered around the day with the highest incident cases from smoothed curves generated through a penalized cubic spline Poisson regression model fit to the incident cases.

Details on the data used by each model can be found in Supplementary file 1, with further details found on the COVID-19 Scenario Modeling Hub GitHub repository website (https://github.com/midas-network/covid19-scenario-modeling-hub; DOI: 10.5281/zenodo.6584489) (COVID-19 Scenario Modeling Hub, 2021). All model output data and ensembled estimates are publicly available on the GitHub repository. All code used to generate numbers and figures reported in this manuscript are publicly available via the GitHub repository. Code required for ensembling model outputs can be made available upon request. Figure, code, and data are available through the open-source MIT license.

Disclaimer: The findings and conclusions in this report are those of the authors and do not necessarily represent the views of the Centers for Disease Control and Prevention or the National Institutes of Health. Any use of trade, firm, or product names is for descriptive purposes only and does not imply endorsement by the US Government.

All model output data are available on the project github at https://github.com/midas-network/covid19-scenario-modeling-hub (archived at https://doi.org/10.5281/zenodo.6584489). Code and data specific to this manuscript has been consolidated into a repository at https://github.com/midas-network/covid19-scenario-modeling-hub/tree/master/paper-source-code/round-7. All data used are publicly available.

1. 1. Contamin L

   (2022) GitHub

   ID midas-network/covid19-scenario-modeling-hub. COVID-19 Scenario Modeling Hub.

   https://github.com/midas-network/covid19-scenario-modeling-hub
2. 1. Smith CP

   (2022) GitHub

   ID midas-network/covid19-scenario-modeling-hub/tree/master/paper-source-code/round-7. Projected resurgence of COVID-19 in the United States in July-December 2021 resulting from the increased transmissibility of the Delta variant and faltering vaccination.

   https://github.com/midas-network/covid19-scenario-modeling-hub/tree/master/paper-source-code/round-7

1. 1. Allen H
   2. Vusirikala A
   3. Flannagan J
   4. Twohig KA
   5. Zaidi A
   6. Chudasama D
   7. Lamagni T
   8. Groves N
   9. Turner C
   10. Rawlinson C
   11. Lopez-Bernal J
   12. Harris R
   13. Charlett A
   14. Dabrera G
   15. Kall M
   16. COVID-19 Genomics UK COG-UK Consortium

   (2022) Household transmission of COVID-19 cases associated with SARS-CoV-2 delta variant (B.1.617.2): national case-control study

   The Lancet Regional Health. Europe 12:100252.

   https://doi.org/10.1016/j.lanepe.2021.100252

   • PubMed
   • Google Scholar
2. 1. Borchering RK
   2. Viboud C
   3. Howerton E
   4. Smith CP
   5. Truelove S
   6. Runge MC
   7. Reich NG
   8. Contamin L
   9. Levander J
   10. Salerno J
   11. van Panhuis W
   12. Kinsey M
   13. Tallaksen K
   14. Obrecht RF
   15. Asher L
   16. Costello C
   17. Kelbaugh M
   18. Wilson S
   19. Shin L
   20. Gallagher ME
   21. Mullany LC
   22. Rainwater-Lovett K
   23. Lemaitre JC
   24. Dent J
   25. Grantz KH
   26. Kaminsky J
   27. Lauer SA
   28. Lee EC
   29. Meredith HR
   30. Perez-Saez J
   31. Keegan LT
   32. Karlen D
   33. Chinazzi M
   34. Davis JT
   35. Mu K
   36. Xiong X
   37. Pastore Y Piontti A
   38. Vespignani A
   39. Srivastava A
   40. Porebski P
   41. Venkatramanan S
   42. Adiga A
   43. Lewis B
   44. Klahn B
   45. Outten J
   46. Schlitt J
   47. Corbett P
   48. Telionis PA
   49. Wang L
   50. Peddireddy AS
   51. Hurt B
   52. Chen J
   53. Vullikanti A
   54. Marathe M
   55. Healy JM
   56. Slayton RB
   57. Biggerstaff M
   58. Johansson MA
   59. Shea K
   60. Lessler J

   (2021) Modeling of Future COVID-19 Cases, Hospitalizations, and Deaths, by Vaccination Rates and Nonpharmaceutical Intervention Scenarios - United States, April-September 2021

   MMWR. Morbidity and Mortality Weekly Report 70:719–724.

   https://doi.org/10.15585/mmwr.mm7019e3

   • PubMed
   • Google Scholar
3. Website

   1. Carnegie Mellon University Delphi Group

   (2021) COVIDcast

   Accessed May 29, 2021.

   https://delphi.cmu.edu/covidcast/indicator/?date=20210703&sensor=fb-survey-smoothed_wcovid_vaccinated_or_accept
4. Software

   1. Centers for Disease Control and Prevention

   (2021a) Guidance for COVID-19 Prevention in K-12 Schools

   Centers for Disease Control and Prevention.

   https://www.cdc.gov/coronavirus/2019-ncov/community/schools-childcare/k-12-guidance.html
5. Website

   1. Centers for Disease Control and Prevention

   (2021b) COVID Data Tracker

   Accessed June 19, 2021.

   https://covid.cdc.gov/covid-data-tracker/#nowcast
6. Software

   1. Coronavirus Resource Center

   (2020) Coronavirus Resource Center

   Johns Hopkins University & Medicine.

   https://coronavirus.jhu.edu/region/mexico
7. Software

   1. COVID-19 Scenario Modeling Hub

   (2020) COVID-19 Scenario Modeling Hub

   COVID-19.

   https://covid19scenariomodelinghub.org/viz.html
8. Software

   1. COVID-19 Scenario Modeling Hub

   (2021) Model Metadata

   COVID-19.

   https://viz.covid19scenariomodelinghub.org/#tab-3721-2
9. Preprint

   1. Elliott P
   2. Haw D
   3. Wang H
   4. Eales O
   5. Walters CE
   6. Ainslie KEC
   7. Atchison C
   8. Fronterre C
   9. Diggle PJ
   10. Page AJ
   11. Trotter AJ
   12. Prosolek SJ
   13. Ashby D
   14. Donnelly CA
   15. Barclay W
   16. Taylor G
   17. Cooke G
   18. Ward H
   19. Darzi A
   20. Riley S
   21. The COVID-19 Genomics UK COG-UK Consortium

   (2021) REACT-1 Round 13 Final Report: Exponential Growth, High Prevalence of SARS-CoV-2 and Vaccine Effectiveness Associated with Delta Variant in England during May to July 2021

   medRxiv.

   https://doi.org/10.1101/2021.09.02.21262979

   • Google Scholar
10. Website

    1. Estimates of Vaccine Hesitancy for COVID-19

    (2021) Centers for Disease Control and Prevention

    Accessed May 29, 2021.

    https://data.cdc.gov/stories/s/Vaccine-Hesitancy-for-COVID-19/cnd2-a6zw
11. Preprint

    1. Higdon MM
    2. Wahl B
    3. Jones CB
    4. Rosen JG
    5. Truelove SA
    6. Baidya A
    7. Nande AA
    8. ShamaeiZadeh PA
    9. Walter KK
    10. Feikin DR
    11. Patel MK
    12. Knoll MD
    13. Hill AL

    (2021) A Systematic Review of COVID-19 Vaccine Efficacy and Effectiveness against SARS-CoV-2 Infection and Disease

    bioRxiv.

    https://doi.org/10.1101/2021.09.17.21263549

    • Google Scholar
12. 1. Jose VRR
    2. Grushka-Cockayne Y
    3. Lichtendahl KC

    (2014) Trimmed Opinion Pools and the Crowd’s Calibration Problem

    Management Science 60:463–475.

    https://doi.org/10.1287/mnsc.2013.1781

    • Google Scholar
13. Preprint

    1. Li B
    2. Deng A
    3. Li K
    4. Hu Y
    5. Li Z
    6. Xiong Q
    7. Liu Z
    8. Guo Q
    9. Zou L
    10. Zhang H
    11. Zhang M
    12. Ouyang F
    13. Su J
    14. Su W
    15. Xu J
    16. Lin H
    17. Sun J
    18. Peng J
    19. Jiang H
    20. Zhou P
    21. Hu T
    22. Luo M
    23. Zhang Y
    24. Zheng H
    25. Xiao J
    26. Liu T
    27. Che R
    28. Zeng H
    29. Zheng Z
    30. Huang Y
    31. Yu J
    32. Yi L
    33. Wu J
    34. Chen J
    35. Zhong H
    36. Deng X
    37. Kang M
    38. Pybus OG
    39. Hall M
    40. Lythgoe KA
    41. Li Y
    42. Yuan J
    43. He J
    44. Lu J

    (2021) Viral Infection and Transmission in a Large, Well-Traced Outbreak Caused by the SARS-CoV-2 Delta Variant

    bioRxiv.

    https://doi.org/10.1101/2021.07.07.21260122

    • Google Scholar
14. Software

    1. NCIRD

    (2021) Centers for Disease Control and Prevention. COVID-19 Vaccination Trends in the United States National and Jurisdictional

    Centers for Disease Control and Prevention.

    https://data.cdc.gov/Vaccinations/COVID-19-Vaccination-Trends-in-the-United-States-N/rh2h-3yt2
15. Preprint

    1. Puranik A
    2. Lenehan P
    3. Silvert E
    4. Niesen MJ
    5. Corchado-Garcia J
    6. O’Horo JC
    7. Virk A
    8. Swift MD
    9. Halamka J
    10. Badley AD
    11. Venkatakrishnan A
    12. Soundararajan V

    (2021) Comparison of Two Highly-Effective MRNA Vaccines for COVID-19 During Periods of Alpha and Delta Variant Prevalence

    medRxiv.

    https://doi.org/10.2139/ssrn.3902782

    • Google Scholar
16. 1. Stone M

    (1961) The Opinion Pool

    The Annals of Mathematical Statistics 32:1339–1342.

    https://doi.org/10.1214/aoms/1177704873

    • Google Scholar
17. Website

    1. US Department of Health and Human Services

    (2020) COVID-19 Reported Patient Impact and Hospital Capacity by State Timeseries

    Accessed May 29, 2021.

    https://healthdata.gov/Hospital/COVID-19-Reported-Patient-Impact-and-Hospital-Capa/g62h-syeh

Decision letter after peer review:

Thank you for submitting your article "Projected resurgence of COVID-19 in the United States in July-December 2021 resulting from the increased transmissibility of the Δ variant and faltering vaccination" for consideration by eLife. Your article has been reviewed by 3 peer reviewers, one of whom is a member of our Board of Reviewing Editors, and the evaluation has been overseen by a Senior Editor. The following individual involved in review of your submission has agreed to reveal their identity: Jodie McVernon (Reviewer #3).

As is customary in eLife, the reviewers have discussed their critiques with one another. What follows below is the Reviewing Editor's edited compilation of the essential and ancillary points provided by reviewers in their critiques and in their interaction post-review. Please submit a revised version that addresses these concerns directly. Although we expect that you will address these comments in your response letter, we also need to see the corresponding revision clearly marked in the text of the manuscript. Some of the reviewers' comments may seem to be simple queries or challenges that do not prompt revisions to the text. Please keep in mind, however, that readers may have the same perspective as the reviewers. Therefore, it is essential that you attempt to amend or expand the text to clarify the narrative accordingly.

Essential revisions:

1. Provide a table describing the basic features of the models included in this analysis. Particular parameters that should be described include vaccine efficacy against infection and assumptions regarding natural immunity to reinfection.

2. Provide an analysis showing individual model predictions for a scenario in order to assess inter-model variability of predictions.

3. Provide further discussion regarding what assumptions of the models may have led to the underestimation of the number of cases in the Summer/Fall, and the features of the models that were more closely able to reproduce the observed resurgence.

4. Comment on the timeliness of predictions and the time horizon over which predictions are likely to be valid.

5. Discussion on whether heterogeneity below the state level could have contributed to the overestimation of vaccination effectiveness by the models.

6. Discussion on the impact of waning vaccine immunity, whether this was accounted for by the models, and whether this will be accounted for in future rounds of predictions.

Reviewer #1 (Recommendations for the authors):

– The current first section "overview" serves as both introduction, and summary of results. I think the manuscript would benefit from a more traditional introduction, where the objectives of the COVID-19 Scenario Modeling Hub are detailed, results from previous rounds are summarized, and the objectives of the current round (6? 7? I am not very clear on this) are detailed. The main results could be summarized in the discussion.

– There is very little information about the models which contributing predictions to this analysis. This does not allow the reader to fully assess the underlying methods. I would generally expect in any multiple model analysis to have a table describing basic features of the models, in the same way that a systematic review/meta-analysis would describe the features of the underlying studies. Key features to describe would be the type of model structure, design, and basic underlying assumptions likely to vary between models such as effectiveness against infection, transmission, and progression to severe outcomes, demographic stratifications included etc.

– The main value of combining different model predictions is to capture and explore uncertainty due to differences between models. Please consider including a figure of cross-model comparison of individual model predictions for a given scenario. Some discussion regarding differences between models leading to differences in predictions would also be warranted.

– I liked the comparison between model predictions and observed data for July. I would have liked to see some discussion relating to how this information is likely to influence future rounds of modelling and predictions, as it suggests that some of the assumptions of the models do not reflect the current epidemiology on the ground.

Reviewer #2 (Recommendations for the authors):

1) About the timeliness of the projections: I received the review invitation on 28 September 2021. Could the authors comment on the timeliness of projections? For example, if six-month projections were not possible, would it be more practical to provide projections every two months?

2) About model assumptions and Table 1: It is not clear to me what vaccine efficacy meant here in Table 1. Vaccine efficacy against infection, symptomatic infection, hospitalization, or death of COVID-19? Since the vaccine uptake by 3 July 2021 was the most important factor in the projections of cumulative cases and deaths in Figure 2, what were the assumptions in the nine models about immunity from vaccination and natural infections from the previous waves? Could the authors summarize the most important assumptions for each of the nine models?

3) About source of uncertainties: There were substantial uncertainties in the model projections in Figure 1 (i.e., the 95% CI of projections could cover nearly all the possible actual outcomes). Could the authors identify the major sources of uncertainties of each of the nine models?

Although all the data and codes are publicly available, it will be great if the authors could add a table to summarize the major similarities and differences of the nine models, given the high uncertainties in projections across different models in Figure 1.

Reviewer #3 (Recommendations for the authors):

This ensemble modelling exercise is incredibly useful, but could be even further enhanced by more of a discussion of difference, as indicated above. It will be very important to include more considerations of immune waning and boosting in future rounds.

Will future rounds incorporate assumptions about the duration of protection following the primary series, particularly as boosters come online and will also need to be incorporated? Could waning be a partial explanation for the overestimation of vaccine impacts here?

Naturally acquired immunity is mentioned but is natural boosting of immunised individuals built into any of these models?

Essential revisions:

1. Provide a table describing the basic features of the models included in this analysis. Particular parameters that should be described include vaccine efficacy against infection and assumptions regarding natural immunity to reinfection.

We have added Supplementary File 1, a table that provides a summary of the assumptions for each of the nine contributing models. We have added the following sentence to the Methods (line 101, pg 6):

“For details on individual model assumptions, see Supplementary File 1.”

2. Provide an analysis showing individual model predictions for a scenario in order to assess inter-model variability of predictions.

We have added Figure4-supplemental figures 1-3, which display the individual model projections, demonstrating the inter-model variability of predictions. We have added a reference to these figures on line (230-231). Further, our new supplemental tables describing individual model assumptions will help readers better understand some of the drivers of inter-model variability (Supplementary File 1).

Although we comment on some of the drivers of this variability in the text (see 4th paragraph of Discussion, pg 11), we believe that a formal analysis of individual model predictions is beyond the scope of this paper. Differences between models are many and include variation in model structure, parameters, and calibration; this understanding the drivers of model variation is not straightforward. Overall, there are many dimensions of uncertainty in the epidemiologic situation, and it is likely that no individual model appropriately captures all these aspects of uncertainty. By combining the models together into an ensemble, we aim to better capture the range of this uncertainty and therefore to provide projections that are more useful for public health decision making.

3. Provide further discussion regarding what assumptions of the models may have led to the underestimation of the number of cases in the Summer/Fall, and the features of the models that were more closely able to reproduce the observed resurgence.

We have added additional text to further detail why projections underestimate the observed resurgences.

In the discussion (pg 10), we now state:

“Possible mechanisms driving the underestimation of the observed Summer 2021 resurgence may include inaccurate assumptions about the transmissibility and severity of Delta (including changes in serial interval or severity of infection relative to other variants), the effectiveness of the vaccines against infection and transmission, heterogeneity in vaccine coverage at the county level or below, waning of natural and vaccine-derived immunity, changes in testing practices, and the interaction of these factors with NPIs and behavior change.^13–15 Assumptions regarding these factors were left to the discretion of the teams and therefore vary across models.”

No individual model was able to successfully reproduce the observed resurgences at the state-level. Those models that tracked with the observed resurgence at the national level for certain scenarios (particularly USC) did not consistently do so at the state-level.

4. Comment on the timeliness of predictions and the time horizon over which predictions are likely to be valid.

These should not be considered predictions, but rather projections of the epidemic trajectory under a predefined set of assumptions (the “scenario”). It is likely that no scenario will perfectly align with reality over the projection period and hence a simple comparison between observations and projections (as in forecast) is not satisfactory. As a case in point, the Omicron variant emerged at the end of our projection period and was not part of the scenarios that were designed in June 2021 (round 7) and are discussed in this paper.

We note that a full evaluation of our scenario projection efforts is underway and includes various stages of the pandemic (not just the Δ wave). We find that accuracy, measured by coverage or other scoring statistics, does not solely depend on time horizon, but also on the stage of the pandemic and on scenario design. In this round of projection particularly, coverage degrades rapidly over the first 9 weeks of the projection period but recovers in subsequent weeks (as can be seen in Figure 1). A full evaluation is beyond the scope of this paper, and in the meantime, we show a figure comparing state-level projections and observations over the first four weeks of projections (Figure 3).

As regards timeliness, our round 7 projections were made publicly available in mid-July 2021, as COVID19 incidence had just started rising, with the peak of the Δ wave occurring in the first week of September 2021. Hence our projections afforded reasonable time for action.

We have added two sentences in conclusion to highlight the timeliness of our projections (pg 13) and note the inherent difficulties in evaluating scenario-based projections and changes in accuracy by time horizon (pg 12). We also comment that projections are a valid tool that can be used to demonstrate the anticipated impact of changes to the epidemiologic situation – in this case, the relative importance of vaccine coverage and the introduction of a new variant (conclusion).

5. Discussion on whether heterogeneity below the state level could have contributed to the overestimation of vaccination effectiveness by the models.

Teams only submit projections at the state and national level, however many of the contributing models are county-level (Columbia, MOBS, UVA-adaptive, JHU APL). Heterogeneity in vaccination coverage at the county-level (or below) not captured by the models due either to spatial structure or lack of county-level data may very well have contributed to the underestimation of the observed resurgences.

We have added a comment to this effect in discussion pg 10.

6. Discussion on the impact of waning vaccine immunity, whether this was accounted for by the models, and whether this will be accounted for in future rounds of predictions.

Waning of both natural and vaccine-derived immunity was left to the discretion of the individual modeling teams. Only one of the 9 teams included waning immunity in their models. At the time of this round of projections, very little was known about the amount and characteristics of SARS-CoV-2 immunity waning. Subsequent SMH rounds do incorporate waning assumptions; these, along with all rounds of SMH projections, can be accessed on the SMH website (see round 13 in particular, https://covid19scenariomodelinghub.org).

We acknowledge this is now likely a limitation of our projections and have added the following to the Discussion section of this paper to discuss this:

“Critically, eight of the nine teams did not include waning of natural or vaccine-derived immunity, an assumption that we now know is incorrect.¹⁶ Absence of waning resulted in a much smaller population susceptible to infection with Delta, thus reducing the overall potential magnitude that could be projected by these models.”

“In addition, these scenarios do not specify considerations of reinfections or breakthrough infections with the Δ variant, the waning of existing immunity, changes in NPIs and behavior, or vaccination among children aged <12 years starting in November 2021, all of which were expected to be important drivers of dynamics in the subsequent months.”

Reviewer #1 (Recommendations for the authors):

– The current first section "overview" serves as both introduction, and summary of results. I think the manuscript would benefit from a more traditional introduction, where the objectives of the COVID-19 Scenario Modeling Hub are detailed, results from previous rounds are summarized, and the objectives of the current round (6? 7? I am not very clear on this) are detailed. The main results could be summarized in the discussion.

We have modified the overview paragraph to have a bit more traditional feel, splitting it into paragraphs and adding the following paragraph about the COVID-19 Scenario Modeling Hub:

“Established in December 2020, the COVID-19 Scenario Modeling Hub is an effort to apply a multiple-model approach to produce six-month projections of the state and national trajectories of cases, hospitalizations, and deaths in the US under defined scenarios.² Scenarios from projection rounds have focused on control measures, vaccination availability and uptake, and emerging variants.^3”

We did leave a summary of the main findings in the “Overview”, as we felt this was appropriate for this short report, but we reduced this to a briefer summary to mirror a more traditional research article format as suggested. Additionally, to make it clear that this paper is focused on Round 7, we have removed all references to Round 6, focusing on Round 7 only.

– There is very little information about the models which contributing predictions to this analysis. This does not allow the reader to fully assess the underlying methods. I would generally expect in any multiple model analysis to have a table describing basic features of the models, in the same way that a systematic review/meta-analysis would describe the features of the underlying studies. Key features to describe would be the type of model structure, design, and basic underlying assumptions likely to vary between models such as effectiveness against infection, transmission, and progression to severe outcomes, demographic stratifications included etc.

We have added a supplemental table (Supplementary File 1) which provide an overview of individual model structure and assumptions.

– The main value of combining different model predictions is to capture and explore uncertainty due to differences between models. Please consider including a figure of cross-model comparison of individual model predictions for a given scenario. Some discussion regarding differences between models leading to differences in predictions would also be warranted.

We have added Figure 1-supplemental figures 1-3 which show the results of the nine individual models. Additionally, we have added a supplemental table (Supplementary File 1) which provide an overview of individual model structure and assumptions. We provide the following commentary in the Discussion section:

“Further, for a given scenario, there is notable variation among individual model projections with regards to both the timing and the magnitude of the resurgence (Figure 1-supplemental figure 1-3). Variation likely reflects differences in model structure, projected vaccine coverage, projected variant growth, and importance of seasonal effects.”

– I liked the comparison between model predictions and observed data for July. I would have liked to see some discussion relating to how this information is likely to influence future rounds of modelling and predictions, as it suggests that some of the assumptions of the models do not reflect the current epidemiology on the ground.

We update the parameters specified in the scenarios each round to align with the most up to date evidence. For example, we have incorporated waning into later rounds (we have just completed round 13 focused on waning). Further, in later rounds, we paid particular attention to estimates of VE specific to the Delta variant that became available later in the year.

Reviewer #2 (Recommendations for the authors):

1) About the timeliness of the projections: I received the review invitation on 28 September 2021. Could the authors comment on the timeliness of projections? For example, if six-month projections were not possible, would it be more practical to provide projections every two months?

This is a good point and we now comment on the timeless of our projections (conclusion p 13):

“Projections of Delta resurgence presented in this paper were made publicly available in early July 2021 ³, two months ahead of the peak of the Delta wave, providing actionable results. There is a trade-off between releasing projections in a timely manner to guide decisions, and projection accuracy and uncertainty that improve with incorporation of recent information.”

We acknowledge there was a longer lag to turn this work into a publication.

We do acknowledge that as the epidemiological situation changes over the projection period, none of the scenarios will be expected to align with reality. However, the goal of these projections is not to predict the future but rather to demonstrate the anticipated impact of changes in vaccination coverage or the introduction of a more transmissible variant. While on occasion, we have provided predictions for shorter horizons (eg two closely spaced rounds of 3-month projections for Omicron, rounds 10 and 11, 17 days apart), these are more appropriate for emergency situations. Overall, we believe that a longer projection period better conveys the long-term impacts of changes in disease dynamics.

2) About model assumptions and Table 1: It is not clear to me what vaccine efficacy meant here in Table 1. Vaccine efficacy against infection, symptomatic infection, hospitalization, or death of COVID-19? Since the vaccine uptake by 3 July 2021 was the most important factor in the projections of cumulative cases and deaths in Figure 2, what were the assumptions in the nine models about immunity from vaccination and natural infections from the previous waves? Could the authors summarize the most important assumptions for each of the nine models?

The vaccine effectiveness specified in the scenarios is against symptomatic disease. We have added a note to table 1 clarifying this point. Further, we have added a supplemental table (Supplementary File 1) which provide an overview of individual model structure and assumptions.

3) About source of uncertainties: There were substantial uncertainties in the model projections in Figure 1 (i.e., the 95% CI of projections could cover nearly all the possible actual outcomes). Could the authors identify the major sources of uncertainties of each of the nine models?

We note in the text several key drivers of uncertainty:

“Uncertainty may arise from three main sources: specification of the scenarios (e.g., uncertainty in transmissibility); errors in the structure or assumptions of individual models given a specific scenario (e.g., variations in assumptions about vaccination uptake); and inaccurate calibration based on incomplete or biased data (e.g., reporting backlogs). None of the four scenarios considered here are likely to precisely reflect the future reality over a six-month period.²”

Uncertainty in the ensemble, which is obtained by a linear opinion pool approach, is driven both by differences between the trajectories of individual models, and uncertainty on the trajectory of individual models. We do feel that because sources of uncertainty are so varied, a discussion of model-specific results and uncertainty is beyond the scope of this paper. The reader can also refer to new tables in the supplement that describe the major assumptions from each model (Supplementary File 1).

Although all the data and codes are publicly available, it will be great if the authors could add a table to summarize the major similarities and differences of the nine models, given the high uncertainties in projections across different models in Figure 1.

We have added a table to the supplement with the main assumptions of each of the contributing models.

Reviewer #3 (Recommendations for the authors):

This ensemble modelling exercise is incredibly useful, but could be even further enhanced by more of a discussion of difference, as indicated above. It will be very important to include more considerations of immune waning and boosting in future rounds.

We have strengthened our discussion of sources of model differences and provided a supplementary table of model assumptions. Subsequent SMH rounds have focused on different scenarios of waning and immune escape, which we now point to in the revised discussion. Results can be found at https://covid19scenariomodelinghub.org/index.html.

Will future rounds incorporate assumptions about the duration of protection following the primary series, particularly as boosters come online and will also need to be incorporated? Could waning be a partial explanation for the overestimation of vaccine impacts here?

This is certainly a possibility for future rounds. Round 8, 11 and round 13 incorporated waning immunity and future rounds may further refine this (round 13 is publicly available here https://covid19scenariomodelinghub.org/index.html). It is possible that improperly accounting for waning contributed to the underestimation of cases during the projection period, which we now mention.

Naturally acquired immunity is mentioned but is natural boosting of immunised individuals built into any of these models?

This was left to the discretion of the teams.

Author details

1. Shaun Truelove

   Johns Hopkins Bloomberg School of Public Health, Johns Hopkins University, Baltimore, United States

   Contribution

   Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Software, Supervision, Validation, Visualization, Writing – original draft, Writing – review and editing

   Contributed equally with

   Claire P Smith, Michael C Runge and Cecile Viboud

   For correspondence

   shauntruelove@jhu.edu

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-0538-0607

2. Claire P Smith

   Johns Hopkins Bloomberg School of Public Health, Johns Hopkins University, Baltimore, United States

   Contribution

   Conceptualization, Data curation, Formal analysis, Visualization, Writing – original draft, Writing – review and editing

   Contributed equally with

   Shaun Truelove, Michael C Runge and Cecile Viboud

   Competing interests

   No competing interests declared

3. Michelle Qin

   Harvard University, Cambridge, Massachusetts, United States

   Contribution

   Data curation, Formal analysis, Visualization, Writing – original draft, Writing – review and editing

   Competing interests

   No competing interests declared

4. Luke C Mullany

   1. Johns Hopkins Bloomberg School of Public Health, Johns Hopkins University, Baltimore, United States
   2. Johns Hopkins University Applied Physics Laboratory, Laurel, United States

   Contribution

   Conceptualization, Data curation, Formal analysis, Software, Visualization, Writing – original draft, Writing – review and editing

   Competing interests

   No competing interests declared

5. Rebecca K Borchering

   Pennsylvania State University, University Park, United States

   Contribution

   Conceptualization, Formal analysis, Methodology, Project administration, Visualization, Writing – original draft, Writing – review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-4309-2913

6. Justin Lessler

   University of North Carolina at Chapel Hill, Chapel Hill, United States

   Contribution

   Conceptualization, Funding acquisition, Methodology, Project administration, Supervision, Visualization, Writing – original draft, Writing – review and editing

   Competing interests

   has served as an expert witness on cases where the likely length of the pandemic was of issue

7. Katriona Shea

   Pennsylvania State University, University Park, United States

   Contribution

   Conceptualization, Funding acquisition, Methodology, Project administration, Supervision, Writing – original draft, Writing – review and editing

   Competing interests

   No competing interests declared

8. Emily Howerton

   Pennsylvania State University, University Park, United States

   Contribution

   Formal analysis, Methodology, Software, Validation, Visualization, Writing – original draft, Writing – review and editing

   Competing interests

   No competing interests declared

9. Lucie Contamin

   University of Pittsburgh, Pittsburgh, United States

   Contribution

   Formal analysis, Methodology, Software, Visualization

   Competing interests

   No competing interests declared

10. John Levander

    University of Pittsburgh, Pittsburgh, United States

    Contribution

    Formal analysis, Methodology

    Competing interests

    No competing interests declared

11. Jessica Kerr

    University of Pittsburgh, Pittsburgh, United States

    Contribution

    Data curation, Project administration, Resources, Visualization

    Competing interests

    No competing interests declared

12. Harry Hochheiser

    University of Pittsburgh, Pittsburgh, United States

    Contribution

    Project administration, Resources, Software, Supervision, Writing – original draft, Writing – review and editing

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0001-8793-9982

13. Matt Kinsey

    Johns Hopkins University Applied Physics Laboratory, Laurel, United States

    Contribution

    Formal analysis, Methodology, Software

    Competing interests

    No competing interests declared

14. Kate Tallaksen

    Johns Hopkins University Applied Physics Laboratory, Laurel, United States

    Contribution

    Formal analysis, Methodology, Software

    Competing interests

    No competing interests declared

15. Shelby Wilson

    Johns Hopkins University Applied Physics Laboratory, Laurel, United States

    Contribution

    Data curation, Methodology, Software

    Competing interests

    No competing interests declared

16. Lauren Shin

    Johns Hopkins University Applied Physics Laboratory, Laurel, United States

    Contribution

    Formal analysis, Methodology

    Competing interests

    No competing interests declared

17. Kaitlin Rainwater-Lovett

    Johns Hopkins University Applied Physics Laboratory, Laurel, United States

    Contribution

    Conceptualization, Funding acquisition, Methodology, Project administration, Supervision, Writing – review and editing

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0002-8707-7339

18. Joseph C Lemairtre

    École polytechnique fédérale de Lausanne, Lausanne, Switzerland

    Contribution

    Formal analysis, Methodology, Software, Writing – review and editing

    Competing interests

    No competing interests declared

19. Juan Dent

    Johns Hopkins Bloomberg School of Public Health, Johns Hopkins University, Baltimore, United States

    Contribution

    Formal analysis, Methodology, Software, Writing – review and editing

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0003-3154-0731

20. Joshua Kaminsky

    Johns Hopkins Bloomberg School of Public Health, Johns Hopkins University, Baltimore, United States

    Contribution

    Formal analysis, Methodology, Software

    Competing interests

    No competing interests declared

21. Elizabeth C Lee

    Johns Hopkins Bloomberg School of Public Health, Johns Hopkins University, Baltimore, United States

    Contribution

    Conceptualization, Data curation, Formal analysis, Funding acquisition, Methodology, Software, Writing – review and editing

    Competing interests

    No competing interests declared

22. Javier Perez-Saez

    Johns Hopkins Bloomberg School of Public Health, Johns Hopkins University, Baltimore, United States

    Contribution

    Formal analysis, Methodology, Software

    Competing interests

    No competing interests declared

23. Alison Hill

    Johns Hopkins Bloomberg School of Public Health, Johns Hopkins University, Baltimore, United States

    Contribution

    Formal analysis, Methodology, Software, Writing – review and editing

    Competing interests

    No competing interests declared

24. Dean Karlen

    University of Victoria, Victoria, Canada

    Contribution

    Formal analysis, Methodology, Writing – review and editing

    Competing interests

    No competing interests declared

25. Matteo Chinazzi

    Northeastern University, Boston, United States

    Contribution

    Formal analysis, Methodology, Software, Writing – review and editing

    Competing interests

    No competing interests declared

26. Jessica T Davis

    Northeastern University, Boston, United States

    Contribution

    Formal analysis, Methodology, Writing – review and editing

    Competing interests

    No competing interests declared

27. Kunpeng Mu

    Northeastern University, Boston, United States

    Contribution

    Formal analysis, Methodology, Software

    Competing interests

    No competing interests declared

28. Xinyue Xiong

    Northeastern University, Boston, United States

    Contribution

    Formal analysis, Methodology, Software

    Competing interests

    No competing interests declared

29. Ana Pastore y Piontti

    Northeastern University, Boston, United States

    Contribution

    Formal analysis, Methodology, Software

    Competing interests

    No competing interests declared

30. Alessandro Vespignani

    Northeastern University, Boston, United States

    Contribution

    Conceptualization, Formal analysis, Funding acquisition, Methodology, Project administration, Supervision, Writing – review and editing

    Competing interests

    No competing interests declared

31. Ajitesh Srivastava

    University of Southern California, Los Angeles, United States

    Contribution

    Conceptualization, Formal analysis, Funding acquisition, Methodology, Writing – review and editing

    Competing interests

    No competing interests declared

32. Przemyslaw Porebski

    University of Virginia, Charlottesville, United States

    Contribution

    Formal analysis, Methodology, Software

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0001-8012-5791

33. Srinivasan Venkatramanan

    University of Virginia, Charlottesville, United States

    Contribution

    Conceptualization, Formal analysis, Methodology, Writing – review and editing

    Competing interests

    No competing interests declared

34. Aniruddha Adiga

    University of Virginia, Charlottesville, United States

    Contribution

    Formal analysis, Methodology, Software

    Competing interests

    No competing interests declared

35. Bryan Lewis

    University of Virginia, Charlottesville, United States

    Contribution

    Conceptualization, Formal analysis, Funding acquisition, Methodology, Project administration, Supervision, Writing – review and editing

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0003-0793-6082

36. Brian Klahn

    University of Virginia, Charlottesville, United States

    Contribution

    Formal analysis, Methodology

    Competing interests

    No competing interests declared

37. Joseph Outten

    University of Virginia, Charlottesville, United States

    Contribution

    Formal analysis, Methodology

    Competing interests

    No competing interests declared

38. Mark Orr

    University of Virginia, Charlottesville, United States

    Contribution

    Formal analysis, Methodology, Software

    Competing interests

    No competing interests declared

39. Galen Harrison

    University of Virginia, Charlottesville, United States

    Contribution

    Formal analysis, Methodology, Software

    Competing interests

    No competing interests declared

40. Benjamin Hurt

    University of Virginia, Charlottesville, United States

    Contribution

    Data curation, Formal analysis, Methodology

    Competing interests

    No competing interests declared

41. Jiangzhuo Chen

    University of Virginia, Charlottesville, United States

    Contribution

    Formal analysis, Methodology, Software

    Competing interests

    No competing interests declared

42. Anil Vullikanti

    University of Virginia, Charlottesville, United States

    Contribution

    Formal analysis, Methodology, Software

    Competing interests

    No competing interests declared

43. Madhav Marathe

    University of Virginia, Charlottesville, United States

    Contribution

    Conceptualization, Formal analysis, Methodology, Supervision

    Competing interests

    No competing interests declared

44. Stefan Hoops

    University of Virginia, Charlottesville, United States

    Contribution

    Formal analysis, Methodology, Software

    Competing interests

    No competing interests declared

45. Parantapa Bhattacharya

    University of Virginia, Charlottesville, United States

    Contribution

    Formal analysis, Methodology, Software

    Competing interests

    No competing interests declared

46. Dustin Machi

    University of Virginia, Charlottesville, United States

    Contribution

    Formal analysis, Methodology, Software

    Competing interests

    No competing interests declared

47. Shi Chen

    University of North Carolina at Charlotte, Charlotte, United States

    Contribution

    Formal analysis, Methodology, Project administration, Software

    Competing interests

    No competing interests declared

48. Rajib Paul

    University of North Carolina at Charlotte, Charlotte, United States

    Contribution

    Formal analysis, Methodology, Software

    Competing interests

    No competing interests declared

49. Daniel Janies

    University of North Carolina at Charlotte, Charlotte, United States

    Contribution

    Formal analysis, Methodology, Software

    Competing interests

    No competing interests declared

50. Jean-Claude Thill

    University of North Carolina at Charlotte, Charlotte, United States

    Contribution

    Formal analysis, Methodology

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0002-6651-8123

51. Marta Galanti

    Columbia University, New York, United States

    Contribution

    Conceptualization, Formal analysis, Methodology, Writing – review and editing

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0002-9060-1250

52. Teresa K Yamana

    Columbia University, New York, United States

    Contribution

    Formal analysis, Methodology, Writing – review and editing

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0001-8349-3151

53. Sen Pei

    Columbia University, New York, United States

    Contribution

    Formal analysis, Methodology, Writing – review and editing

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0002-7072-2995

54. Jeffrey L Shaman

    Columbia University, New York, United States

    Contribution

    Formal analysis, Funding acquisition, Methodology, Software, Writing – review and editing

    Competing interests

    and Columbia University disclose partial ownership of SK Analytics. Discloses consulting for BNI

    "This ORCID iD identifies the author of this article:" 0000-0002-7216-7809

55. Jessica M Healy

    CDC COVID-19 Response Team, Atlanta, United States

    Contribution

    Conceptualization, Funding acquisition, Project administration, Supervision, Writing – review and editing

    Competing interests

    No competing interests declared

56. Rachel B Slayton

    CDC COVID-19 Response Team, Atlanta, United States

    Contribution

    Conceptualization, Funding acquisition, Project administration, Resources, Writing – review and editing

    Competing interests

    No competing interests declared

57. Matthew Biggerstaff

    CDC COVID-19 Response Team, Atlanta, United States

    Contribution

    Conceptualization, Funding acquisition, Project administration, Supervision, Writing – review and editing

    Competing interests

    No competing interests declared

58. Michael A Johansson

    CDC COVID-19 Response Team, Atlanta, United States

    Contribution

    Conceptualization, Funding acquisition, Project administration, Supervision

    Competing interests

    No competing interests declared

59. Michael C Runge

    United States Geological Survey, Laurel, United States

    Contribution

    Conceptualization, Formal analysis, Methodology, Project administration, Supervision, Writing – original draft, Writing – review and editing

    Contributed equally with

    Shaun Truelove, Claire P Smith and Cecile Viboud

    Competing interests

    reports stock ownership in Becton Dickinson & Co, which manufactures medical equipment used in COVID testing, vaccination, and treatment

    "This ORCID iD identifies the author of this article:" 0000-0002-8081-536X

60. Cecile Viboud

    Fogarty International Center, National Institutes of Health, Bethesda, United States

    Contribution

    Conceptualization, Formal analysis, Investigation, Methodology, Project administration, Supervision, Writing – original draft, Writing – review and editing

    Contributed equally with

    Shaun Truelove, Claire P Smith and Michael C Runge

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0003-3243-4711

• 1,639

  Page views

• 202

  Downloads

• 14

  Citations

Article citation count generated by polling the highest count across the following sources: PubMed Central, Crossref, Scopus.