As the official death toll of the coronavirus disease 2019 (COVID-19) continues to grow, the full impacts of the pandemic on a range of conditions remain debated. By the end of January 2023, official statistics reported 1,056,000 deaths in the United States alone (Johns Hopkins University, 2022), although burden estimates that do not rely on official tallies suggest a higher death toll (Weinberger et al., 2020; Karlinsky and Kobak, 2021). A pandemic of the magnitude of COVID-19 would be expected to have secondary effects on unrelated health conditions; for instance, non-COVID-19 deaths increased in Spring of 2020 at the height of the first wave in part due to avoidance of the healthcare system (Bollmann et al., 2020; Kansagra et al., 2020; Mafham et al., 2020; Woolf et al., 2020; Woolf et al., 2021).

Excess mortality approaches have been used for over a century to capture the full scope of large-scale infectious disease events, heatwaves, and earthquakes, by measuring the rise in mortality over a historical baseline (Serfling, 1963; Weinberger et al., 2020). In the early phase of the pandemic, these approaches highlighted substantial underestimation in official statistics of COVID-19 deaths due to limited viral testing (Weinberger et al., 2020; Kobak, 2021). More recent analyses have examined excess mortality patterns for specific causes of death by age and socio-demographic groups and have compared the COVID-19 death toll between countries (Banerjee et al., 2020; Islam et al., 2021; Karlinsky and Kobak, 2021; Mena et al., 2021; Rossen et al., 2021; Woolf et al., 2021; COVID-19 Excess Mortality Collaborators, 2022). Yet, separating the direct impact of SARS-CoV-2 infection on mortality from the other consequences of the pandemic remains challenging.

To measure the direct and indirect effects of the COVID-19 pandemic, it is important to enumerate the mechanisms that could generate these effects and understand how they would manifest in mortality statistics. We first consider the direct effects of COVID-19 as those deaths that resulted from SARS-CoV-2 infection and its complications. When there is evidence of SARS-CoV-2 infection in the days or weeks before death, either virologically or clinically, it is likely that these deaths will receive a COVID-19 code during certification and these deaths will appear in official statistics. We would, however, expect variation between states in the death certification process for COVID-19. Additionally, a number of deaths triggered by SARS-CoV-2 infection could result from a complicated and protracted pathologic process, especially in patients with multiple underlying chronic conditions, who may lack a history of SARS-CoV-2 testing, and whose death may not be ascribed to COVID-19. For instance, a death in a diabetic patient could have been triggered by an undetected SARS-CoV-2 infection, resulting in a primary code of diabetes, with a SARS-CoV-2 code either lacking or listed as contributing condition. We would then expect a rise in diabetes mortality to coincide with a rise in COVID-19 cases. A similar phenomenon has been reported for influenza, with mortality from chronic conditions rising concomitantly with influenza-associated respiratory mortality in epidemic and pandemic seasons (Reichert et al., 2004; Quandelacy et al., 2014).

In addition to the direct impacts of COVID-19, there will be positive and negative changes in mortality during the pandemic period that are not associated with SARS-CoV-2 infection and its complications. We refer to these changes as indirect impacts. Reasons for these changes include avoidance of the healthcare system for treatment of acute conditions and for management of underlying chronic conditions, stressed healthcare systems in a period of high COVID-19 incidence, mental health issues in families of patients severely affected by COVID-19, societal disruptions (Sharma et al., 2021), decreased social interaction that depresses circulation of endemic pathogens, and decreased air pollution. The indirect impacts of the pandemic on mental health, violence, and addiction remain particularly debated, with potentially large impacts on mortality (Faust et al., 2021b; Faust et al., 2021c). These indirect mortality changes may or may not coincide temporally with COVID-19 waves.

In the United States, there was substantial geographic and temporal heterogeneity in the trajectory of the COVID-19 pandemic, along with differences in the strength and types of interventions implemented to mitigate COVID-19. In a large country with standardized death ascertainment like the United States, these heterogeneities provide an opportunity to separate the contributions of viral infection from other drivers of mortality. Here, we apply time series approaches to four large waves of COVID-19 from March 1, 2020 to January 1, 2022, to separate the direct consequences of SARS-CoV-2 infection on age- state- and cause-specific mortality from the indirect consequences associated with hospital strain and interventions. Our analyses indicate that the direct and indirect effects of the pandemic vary substantially by chronic condition and age group. A better understanding of these effects is particularly important for the mitigation of future large-scale pandemics.

In this US study, we aimed to disentangle the direct and indirect mortality impacts of the COVID-19 pandemic from March 1, 2020 to January 1, 2022 using regression models and synchronicity analyses. We find that 84% (65–94%) of the rise in all-cause mortality during this period can be statistically linked to SARS-CoV-2 activity, lending support to the predominance of the direct mortality consequences of the pandemic on a national scale. We also find a direct contribution of SARS-CoV-2 infection to mortality from several chronic conditions such as Alzheimer’s, diabetes, and heart diseases. This contribution is not captured in official statistics that consider COVID-19 as the primary cause of death. In contrast, analysis of mortality in children and young adults, and mortality from accidents and injuries, drug overdoses, assaults, and homicides, paints a different picture. Modeling of these death strata indicates a marked relationship with proxies for the strength of interventions, supporting a dominant contribution of indirect pandemic effects unrelated to SARS-CoV-2 infection. In contrast to other causes of death studied, cancer and suicides remained within baseline levels during the pandemic period.

Perhaps the most striking finding of our study is the large mortality burden of the pandemic in individuals 25–44 years, with an estimated 112,200 (100,200–123,100) excess deaths by January 1, 2022. Only 30% of these excess deaths are ascribed to COVID-19 in official statistics. Accordingly, our regression analysis does not support a predominant contribution of SARS-CoV-2 infection in this age group. The trajectory of mortality in this age group is disjoint from periods of intense COVID-19 circulation and statistically tied to a variable monitoring the strength of interventions. This finding supports a possible detrimental effect of COVID-19 control measures beyond the initial lockdown period in Spring 2020, although this is an ecological study that cannot prove causality nor elucidate the mechanisms at play. And while individuals under 25 years had a low overall excess death rate during the pandemic, we find that the contribution of indirect pandemic effects is even greater in this age group. In contrast, individuals over 65 years predominantly suffered from the direct consequences of SARS-CoV-2 infection. In a study of excess mortality in over 100 countries, Karlinsky and Kobak, 2021 note a predominance of the direct mortality consequences of the pandemic; however, they did not study age patterns. Our analysis shows that the contributions of direct and indirect effects vary with age and can skew towards indirect effects in the young, even in countries that experienced relatively high infections rates like the United States.

Prior studies have shown that a decrease in emergency visits for diabetes, stroke, and myocardial infarctions in all age groups coincided with a rise in mortality for these conditions (Lange et al., 2020). Faust et al., estimated that 38% of deaths between 25 and 44 years were due to COVID-19 during March to July 2020, compared to 26% (23–30%) in our study that considers a much longer time period (Faust et al., 2021a). Public health interventions, limited medical care, and behavioral changes (e.g. delays in seeking timely medical help due to fear of infection, Bollmann et al., 2020; Kansagra et al., 2020; Mafham et al., 2020; Woolf et al., 2021) could have contributed to the surge in excess deaths unrelated to COVID-19 in young adults, resulting in a notable peak of mortality in summer 2020. In addition, we find that mortality from external causes remained elevated during May to August 2020 among young adults, likely driven by an elevation in deaths from opioid poisonings, accidents, and assaults.

Mortality from external causes increased by 102,800 (81,400–123,700) from March 1, 2020 to January 1, 2022. There was a moderate correlation with pre-pandemic baseline mortality rates in state-level data (Appendix 1—figure 13), indicating that states with historically high death rates from external causes experienced more prominent increases during the pandemic. The rise in external cause mortality was most pronounced in the subcategory of assaults and homicide, followed by overdoses and accidents. In contrast, mortality from suicides remained stable or below expectations. Prior work by Faust et al., 2021b, reported a decrease in suicide in several countries during March to July 2020, including in the United States (10%); here, we show that suicide mortality remained stable throughout the rest of 2020–2021. Furthermore, Faust et al., 2021a reported that deaths from overdoses and injuries increased during March-July 2020. In our data, the increase persisted until the end of 2021. Overall, of the eight causes of death studied here, the indirect pandemic effects were statistically largest in external mortality causes.

There was strong synchronicity between respiratory mortality and mortality from other conditions during spring 2020, possibly due to poor SARS-CoV-2 test availability and guidelines to restrict testing to just hospitalized cases in the early pandemic stages. Many deaths in nursing homes or at home during March to April 2020 were never tested, and they were recorded as known unlying conditions (i.e. heart disease, Alzheimer’s, and diabetes) by default. In addition, Alzheimer patients typically live in long-term care facilities and may have been at increased risk of (untested) COVID-19 infection early in the pandemic. Interestingly, the correlation between excess mortality from respiratory diseases and Alzheimer increased in the winter 2020–2021 wave, signaling a persistent direct impact of COVID-19 on Alzheimer’s in a period where COVID-19 incidence and testing propensity were high.

We validated our excess mortality estimates against serology and assessed the IFR, a parameter notoriously difficult to measure. Our all-age estimate of 0.67% (95% CI 0.60–0.73%) is consistent with a 2020 meta-analysis (Meyerowitz-Katz and Merone, 2020) and an early study from China (0.66%; 95% CI: 0.39—1.33%) (Verity et al., 2020). A study of all-cause excess mortality in the Netherlands reports a substantially higher IFR (1%) (van Asten et al., 2021); however, all-cause mortality is not specific to COVID-19 (accordingly, our estimate based on all-cause mortality is higher at 0.89% (95% CI 0.77–1.02%)). IFR estimates based on official COVID-19 statistics were 15% higher than those based on excess respiratory mortality, yet the official statistics did not correlate with serology data as well as with our excess mortality estimates. Between-state differences in COVID-19 death coding practices could explain these findings. Overall, our analyses support the robustness and specificity of excess respiratory mortality (with the addition of the COVID-19 specific code) as an indicator of the COVID-19 mortality burden.

A few states were outliers in the regression of excess mortality against serology; for instance, New York had a higher than predicted IFR, while Michigan had a lower than predicted IFR. Several non-mutually exclusive factors could drive these findings, including a higher proportion of deaths among older individuals (aligned with the demography of New York state), large outbreaks in long-term care facilities, lack of knowledge on the management of severe patients early in the pandemic (Barnett et al., 2020), and waning immunity. Serosurveys conducted in December 2021 could underestimate cumulative SARS-CoV-2 infection rates in states that have experienced most of their infections in early 2020. All states contributing to CDC serology surveillance used the Roche assay to test for the presence of SARS-CoV-2 antibodies, which is less prone to waning than other assays but has some decay (Prete et al., 2022). We ran a sensitivity analysis using the maximum seroprevalence recorded over the study period rather than the seroprevalence at the end of the study period (Appendix 1—figure 10); New York remained an outlier in this analysis.

The roll-out of a large SARS-CoV-2 vaccination campaign starting in December 2020 in the United States has had a major impact on rates of hospitalizations and deaths for COVID-19. Yet, excess mortality from all causes and respiratory diseases has remained elevated all throughout 2021, fueled by new SARS-CoV-2 variants, particularly Delta and Omicron. COVID-19 mortality is now concentrated among unvaccinated groups, with the highest vaccination rates reported in older individuals (Centers for Disease Control and Prevention, 2022h). Coincidentally, in our data, excess respiratory mortality started to decouple from chronic condition excess mortality (e.g. diabetes, Alzheimer, heart diseases) around the time of the Delta wave in the summer of 2021. Similarly, among the oldest and most highly vaccinated age groups, the Delta wave had a proportionally milder impact than the wave dominated by the ancestral strain in winter 2020–21, while the pattern was reversed in younger age groups. Decoupling between COVID-19 mortality and mortality from chronic conditions is likely due to vaccination rather than a specific variant. Yet decoupling remains insufficient to obfuscate synchronicity in these conditions observed during the full study period from March 1, 2020 to January 1, 2022. Further analyses of the Omicron period could reveal different synchronicity patterns.

It is interesting that during April-June 2021, and before the rise of the Delta variant, mortality from cancer, Alzheimer and heart diseases was below baseline (negative excess mortality), with a similar phenomenon observed for all-cause mortality among individuals 75–84 and over 85 years. These negative excesses could signal a displacement of the mortality baseline, whereby frail individuals are harvested by a large-scale infectious disease event, resulting in a decline in baseline mortality in the aftermath -- similar patterns have been associated with heatwaves (Saha et al., 2014). Harvesting is also consistent with our regression analysis, where estimates of the direct impacts of COVID-19 exceeded 100% in the two oldest age groups (albeit with broad confidence intervals, Table 4). This would be expected if the baseline mortality was overestimated due to harvesting. Furthermore, the age profile of COVID-19 severity risk dictates that older individuals would bear the strongest effects of harvesting, which is consistent with the US data. A competing hypothesis for an inflated baseline is the role of non-COVID-19 pathogens that were repressed during the pandemic due to social distancing. These pathogens are implicitly included in our baseline model calibrated to pre-pandemic years. Although the timing of negative excesses (Spring 2021) is not fully supportive of the contribution of missed pathogens which tend to predominate in winter, harvesting and depressed pathogen circulation are two mechanisms inherently difficult to separate. A third competing mechanism is that official COVID-19 deaths capture deaths with COVID-19, rather than deaths from COVID-19. Although this is likely true to some extent, our estimates of direct effects exceed 100% in regression models that ignore official statistics, suggesting that other mechanisms are at play.

Our study is subject to several limitations. First, mortality counts below the minimum cut-off value of 10 were suppressed due to privacy regulations. As a result, our age-specific analyses were restricted to larger states, and we could not assess the role of race and ethnicity. Prior work has shown important disparities in COVID-19 impact by race/ethnicity and economic status in the United States and abroad (Mena et al., 2021; Rossen et al., 2021). Second, official coding practices may have changed between states and through time-based on SARS-CoV-2 testing availability, location of death, demographic factors, and comorbidities. Third, we assumed full coverage of death reporting, which may not be valid throughout the United States (Murray et al., 2010), and we did not study changes in deaths ascribed to ill-defined codes (R codes). Ill-defined deaths would be captured in all-cause mortality but not in cause-specific analyses. Fourth, we find periods of negative excesses in cancer (throughout the pandemic), cardiovascular, and heart diseases, possibly due to changes in the ascertainment of the underlying cause of death (e.g. death in a cancer patient with COVID-19 is ascribed to COVID-19), harvesting (Saha et al., 2014), or depressed circulation of endemic pathogens other than influenza. Fifth, we choose to fit the model baseline to data for 2014–2020, which is arbitrary. We studied the sensitivity of our excess mortality estimates to this assumption (Appendix 1 - Figure 9). While national analyses were robust to this choice, as well as state-level analyses of most conditions, state-specific estimates for Alzheimer’s disease were more sensitive. Furthermore, the model baseline did not account for possible point-in-time disasters that may have occurred during the pandemic but are independent of COVID-19 (e.g. a hurricane) or changes in air pollution. Sixth, GAM indicate that the Oxford stringency index, used as a proxy for the strength of interventions, is a dominant predictor of excess mortality from external causes and all-cause mortality in individuals under 45 years. Yet, the relationships are non-linear, and the resulting models do not fully capture mortality changes during the pandemic. Along the same lines, the Oxford stringency index does not consider the actual implementation nor the effect of interventions; it is solely based on mandates in place in different locations and time periods. We also assume that, for a given level of stringency, the impact of interventions does not change over time. Because time and intervention stringency are highly conflated, it would be difficult to study potential temporal variation in this relationship. Furthermore, analyses are aggregated at the state or national level, while implementation of interventions may operate more locally. We also do not account for underlying differences in vulnerability between states, where more vulnerable states may have implemented stricter interventions (although this potential bias would not affect temporal analyses). Finally, our study ends on January 1, 2022 and does not capture a recrudescence of COVID-19-related deaths due to the Omicron variant. As a result, our excess mortality estimates should be deemed conservative.

Pandemic excess mortality patterns have been heterogeneous globally (Islam et al., 2021; Karlinsky and Kobak, 2021; Kontis et al., 2020; Nørgaard et al., 2021). In a comprehensive analysis of mortality in 21 countries in Europe, New Zealand, and Australia, official COVID-19 deaths accounted for an average of 77% (62–93%) of all-cause excess deaths during the first wave of the pandemic from March to May 2020 (Kontis et al., 2020). However, there was a greater disconnect between estimates in hard-hit countries such as the UK, Spain, Italy, and Belgium (Kontis et al., 2020; Kontopantelis et al., 2021; Odone et al., 2021). In a study of over 100 countries, the ratio between excess mortality and official deaths was 1.6 on average, but went as high as 50 (Karlinsky and Kobak, 2021). The disconnect was primarily attributed to the under-detection of COVID-19 rather than indirect effects, although indirect effects were not explicitly modeled. In Italy, the case fatality rate for acute myocardial infarction increased threefold during the first wave, while hospitalizations for these conditions decreased by 48% (Odone et al., 2021), suggesting that at least some of the excess deaths were indirect deaths. In the UK, cancer deaths increased about 10% at the height of the April 2020 lockdown; however, more recent fluctuations in cancer mortality remain unclear (Lai et al., 2020). Interestingly, in New Zealand, where control of COVID-19 has been remarkable, mortality was slightly but not significantly below baseline (Kontis et al., 2020). A similar finding was described in Russian provinces where a lockdown was implemented before the onset of COVID-19 (Kobak, 2021). This suggests that a lockdown without COVID-19 is neither preventing nor causing an appreciable number of deaths, although effects could be country-dependent. In the United States, the direct impacts of the pandemic greatly outweigh the indirect consequences in all age data, but the reverse is true in children and young adults. Further work should concentrate on comparing the direct and indirect impact of COVID-19 in different countries over the same time period and using the same methodology.

Our study presents an analysis of publicly available surveillance and mortality data, so no new data have been generated for this manuscript. The modelling code and underlying data have been posted in the following public Github repository https://github.com/viboudc/DirectIndirectCOVID19MortalityEstimation (copy archived at swh:1:rev:f0c23c87a00f589179af5351e89732f51d5a2b19).

1. 1. Centers for Disease Control and Prevention, National Center for Health Statistics

   (2022) National Center for Health Statistics

   ID Weekly-Counts-of-Deaths-by-State-and-Select-Causes/3yf8-kanr. Weekly Counts of Deaths by State and Select Causes, 2014-2019.

   https://data.cdc.gov/NCHS/Weekly-Counts-of-Deaths-by-State-and-Select-Causes/3yf8-kanr
2. 1. Centers for Disease Control and Prevention, National Center for Health Statistics

   (2022) National Center for Health Statistics

   ID Weekly-Counts-of-Deaths-by-State-and-Select-Causes/muzy-jte6. Weekly Counts of Deaths by State and Select Causes, 2020-2023.

   https://data.cdc.gov/NCHS/Weekly-Counts-of-Deaths-by-State-and-Select-Causes/muzy-jte6
3. 1. Centers for Disease Control and Prevention, National Center for Health Statistics

   (2022) National Center for Health Statistics

   ID Provisional-COVID-19-Deaths-by-Week-Sex-and-Age/vsak-wrfu. Provisional COVID-19 Deaths by Week, Sex, and Age.

   https://data.cdc.gov/NCHS/Provisional-COVID-19-Deaths-by-Week-Sex-and-Age/vsak-wrfu
4. 1. Centers for Disease Control and Prevention, National Center for Health Statistics

   (2022) National Center for Health Statistics

   ID Monthly-Counts-of-Deaths-by-Select-Causes-2014-201/bxq8-mugm. Monthly Counts of Deaths by Select Causes, 2014-2019.

   https://data.cdc.gov/NCHS/Monthly-Counts-of-Deaths-by-Select-Causes-2014-201/bxq8-mugm
5. 1. Centers for Disease Control and Prevention, National Center for Health Statistics

   (2022) National Center for Health Statistics

   ID Monthly-Provisional-Counts-of-Deaths-by-Select-Cau/9dzk-mvmi. Monthly Provisional Counts of Deaths by Select Causes, 2020-2022.

   https://data.cdc.gov/NCHS/Monthly-Provisional-Counts-of-Deaths-by-Select-Cau/9dzk-mvmi
6. 1. Centers for Disease Control and Prevention, National Center for Health Statistics

   (2022) National Center for Health Statistics

   ID AH-Monthly-Provisional-Counts-of-Deaths-by-Age-Gro/ezfr-g6hf. AH Monthly Provisional Counts of Deaths by Age Group and HHS region for Select Causes of Death, 2019-2022.

   https://data.cdc.gov/NCHS/AH-Monthly-Provisional-Counts-of-Deaths-by-Age-Gro/ezfr-g6hf
7. 1. Centers for Disease Control and Prevention, National Center for Health Statistics

   (2022) Centers for Disease Control

   ID WONDER. State Population Projections 2004-2030.

   https://wonder.cdc.gov/population-projections.html
8. 1. Centers for Disease Control

   (2022) Centers for Disease Control

   ID covid-data-tracker/#national-lab. Nationwide COVID-19 Infection-Induced Antibody Seroprevalence (Commercial laboratories).

   https://covid.cdc.gov/covid-data-tracker/#national-lab
9. 1. Centers for Disease Control

   (2022) Centers for Disease Control

   ID covid-data-tracker/#datatracker-home. COVID data tracker.

   https://covid.cdc.gov/covid-data-tracker/#datatracker-home

1. 1. Banerjee A
   2. Pasea L
   3. Harris S
   4. Gonzalez-Izquierdo A
   5. Torralbo A
   6. Shallcross L
   7. Noursadeghi M
   8. Pillay D
   9. Sebire N
   10. Holmes C
   11. Pagel C
   12. Wong WK
   13. Langenberg C
   14. Williams B
   15. Denaxas S
   16. Hemingway H

   (2020) Estimating excess 1-year mortality associated with the COVID-19 pandemic according to underlying conditions and age: a population-based cohort study

   Lancet 395:1715–1725.

   https://doi.org/10.1016/S0140-6736(20)30854-0

   • PubMed
   • Google Scholar
2. 1. Barnett ML
   2. Hu L
   3. Martin T
   4. Grabowski DC

   (2020) Mortality, admissions, and patient census at snfs in 3 US cities during the COVID-19 pandemic

   JAMA 324:507–509.

   https://doi.org/10.1001/jama.2020.11642

   • PubMed
   • Google Scholar
3. 1. Bollmann A
   2. Hohenstein S
   3. Meier-Hellmann A
   4. Kuhlen R
   5. Hindricks G

   (2020) Emergency hospital admissions and interventional treatments for heart failure and cardiac arrhythmias in germany during the covid-19 outbreak: insights from the german-wide helios hospital network

   European Heart Journal. Quality of Care & Clinical Outcomes 6:221–222.

   https://doi.org/10.1093/ehjqcco/qcaa049

   • PubMed
   • Google Scholar
4. Website

   1. Centers for Disease Control and Prevention

   (2022a) Weekly Counts of Deaths by State and Select Causes, 2014-2019

   Accessed October 28, 2022.

   https://data.cdc.gov/NCHS/Weekly-Counts-of-Deaths-by-State-and-Select-Causes/3yf8-kanr
5. Website

   1. Centers for Disease Control and Prevention

   (2022b) Weekly Counts of Deaths by State and Select Causes, 2020-2022

   Accessed October 28, 2022.

   https://data.cdc.gov/NCHS/Weekly-Counts-of-Deaths-by-State-and-Select-Causes/muzy-jte6
6. Website

   1. Centers for Disease Control and Prevention

   (2022c) Provisional COVID-19 Deaths by Week, Sex, and Age

   Accessed October 28, 2022.

   https://data.cdc.gov/NCHS/Provisional-COVID-19-Deaths-by-Week-Sex-and-Age/vsak-wrfu
7. Website

   1. Centers for Disease Control and Prevention

   (2022d) Monthly Counts of Deaths by Select Causes, 2014-2019

   Accessed October 28, 2022.

   https://data.cdc.gov/NCHS/Monthly-Counts-of-Deaths-by-Select-Causes-2014-201/bxq8-mugm
8. Website

   1. Centers for Disease Control and Prevention

   (2022e) AH Monthly Provisional Counts of Deaths by Age Group and HHS region for Select Causes of Death, 2019-2022

   Accessed October 28, 2022.

   https://data.cdc.gov/NCHS/AH-Monthly-Provisional-Counts-of-Deaths-by-Age-Gro/ezfr-g6hf
9. Website

   1. Centers for Disease Control and Prevention

   (2022f) State Population Projections 2004-2030

   Accessed October 28, 2022.

   https://wonder.cdc.gov/population-projections.html
10. Website

    1. Centers for Disease Control and Prevention

    (2022g) Nationwide COVID-19 Infection-Induced Antibody Seroprevalence (Commercial laboratories)

    Accessed October 28, 2022.

    https://covid.cdc.gov/covid-data-tracker/#national-lab
11. Website

    1. Centers for Disease Control and Prevention

    (2022h) COVID data tracker

    Accessed October 28, 2022.

    https://covid.cdc.gov/covid-data-tracker/#datatracker-home
12. 1. COVID-19 Excess Mortality Collaborators

    (2022) Estimating excess mortality due to the COVID-19 pandemic: a systematic analysis of COVID-19-related mortality, 2020–21

    Lancet 339:1513–1536.

    https://doi.org/10.1016/S0140-6736(21)02796-3

    • Google Scholar
13. 1. Faust JS
    2. Du C
    3. Mayes KD
    4. Li SX
    5. Lin Z
    6. Barnett ML
    7. Krumholz HM

    (2021a) Mortality from drug overdoses, homicides, unintentional injuries, motor vehicle crashes, and suicides during the pandemic, march-august 2020

    JAMA 326:84–86.

    https://doi.org/10.1001/jama.2021.8012

    • PubMed
    • Google Scholar
14. 1. Faust JS
    2. Krumholz HM
    3. Du C
    4. Mayes KD
    5. Lin Z
    6. Gilman C
    7. Walensky RP

    (2021b) All-cause excess mortality and COVID-19-related mortality among US adults aged 25-44 years, march-july 2020

    JAMA 325:785–787.

    https://doi.org/10.1001/jama.2020.24243

    • PubMed
    • Google Scholar
15. 1. Faust JS
    2. Shah SB
    3. Du C
    4. Li SX
    5. Lin Z
    6. Krumholz HM

    (2021c) Suicide deaths during the COVID-19 stay-at-home advisory in massachusetts, march to may 2020

    JAMA Network Open 4:e2034273.

    https://doi.org/10.1001/jamanetworkopen.2020.34273

    • PubMed
    • Google Scholar
16. 1. Goldstein E
    2. Viboud C
    3. Charu V
    4. Lipsitch M

    (2012) Improving the estimation of influenza-related mortality over a seasonal baseline

    Epidemiology 23:829–838.

    https://doi.org/10.1097/EDE.0b013e31826c2dda

    • PubMed
    • Google Scholar
17. Website

    1. Health and Human Services

    (2022) COVID-19 Reported Patient Impact and Hospital Capacity by State Timeseries

    Accessed August 1, 2022.

    https://healthdata.gov/Hospital/COVID-19-Reported-Patient-Impact-and-Hospital-Capa/g62h-syeh
18. 1. Islam N
    2. Shkolnikov VM
    3. Acosta RJ
    4. Klimkin I
    5. Kawachi I
    6. Irizarry RA
    7. Alicandro G
    8. Khunti K
    9. Yates T
    10. Jdanov DA
    11. White M
    12. Lewington S
    13. Lacey B

    (2021) Excess deaths associated with covid-19 pandemic in 2020: age and sex disaggregated time series analysis in 29 high income countries

    BMJ 373:n1137.

    https://doi.org/10.1136/bmj.n1137

    • PubMed
    • Google Scholar
19. Website

    1. Johns Hopkins University

    (2022) COVID-19 Dashboard: Johns Hopkins Coronavirus Resource Center

    Accessed February 1, 2022.

    https://coronavirus.jhu.edu/data
20. 1. Kansagra AP
    2. Goyal MS
    3. Hamilton S
    4. Albers GW

    (2020) Collateral effect of covid-19 on stroke evaluation in the united states

    The New England Journal of Medicine 383:400–401.

    https://doi.org/10.1056/NEJMc2014816

    • PubMed
    • Google Scholar
21. 1. Karlinsky A
    2. Kobak D

    (2021) Tracking excess mortality across countries during the COVID-19 pandemic with the world mortality dataset

    eLife 10:e69336.

    https://doi.org/10.7554/eLife.69336

    • PubMed
    • Google Scholar
22. 1. Kobak D

    (2021) Excess mortality reveals covid’s true toll in russia

    Significance 18:16–19.

    https://doi.org/10.1111/1740-9713.01486

    • PubMed
    • Google Scholar
23. 1. Kontis V
    2. Bennett JE
    3. Rashid T
    4. Parks RM
    5. Pearson-Stuttard J
    6. Guillot M
    7. Asaria P
    8. Zhou B
    9. Battaglini M
    10. Corsetti G
    11. McKee M
    12. Di Cesare M
    13. Mathers CD
    14. Ezzati M

    (2020) Magnitude, demographics and dynamics of the effect of the first wave of the COVID-19 pandemic on all-cause mortality in 21 industrialized countries

    Nature Medicine 26:1919–1928.

    https://doi.org/10.1038/s41591-020-1112-0

    • PubMed
    • Google Scholar
24. 1. Kontopantelis E
    2. Mamas MA
    3. Deanfield J
    4. Asaria M
    5. Doran T

    (2021) Excess mortality in england and wales during the first wave of the COVID-19 pandemic

    Journal of Epidemiology and Community Health 75:213–223.

    https://doi.org/10.1136/jech-2020-214764

    • PubMed
    • Google Scholar
25. 1. Lai AG
    2. Pasea L
    3. Banerjee A
    4. Hall G
    5. Denaxas S
    6. Chang WH
    7. Katsoulis M
    8. Williams B
    9. Pillay D
    10. Noursadeghi M
    11. Linch D
    12. Hughes D
    13. Forster MD
    14. Turnbull C
    15. Fitzpatrick NK
    16. Boyd K
    17. Foster GR
    18. Enver T
    19. Nafilyan V
    20. Humberstone B
    21. Neal RD
    22. Cooper M
    23. Jones M
    24. Pritchard-Jones K
    25. Sullivan R
    26. Davie C
    27. Lawler M
    28. Hemingway H

    (2020) Estimated impact of the COVID-19 pandemic on cancer services and excess 1-year mortality in people with cancer and multimorbidity: near real-time data on cancer care, cancer deaths and a population-based cohort study

    BMJ Open 10:e043828.

    https://doi.org/10.1136/bmjopen-2020-043828

    • PubMed
    • Google Scholar
26. 1. Lange SJ
    2. Ritchey MD
    3. Goodman AB
    4. Dias T
    5. Twentyman E
    6. Fuld J
    7. Schieve LA
    8. Imperatore G
    9. Benoit SR
    10. Kite-Powell A
    11. Stein Z
    12. Peacock G
    13. Dowling NF
    14. Briss PA
    15. Hacker K
    16. Gundlapalli AV
    17. Yang Q

    (2020) Potential indirect effects of the COVID-19 pandemic on use of emergency departments for acute life-threatening conditions - united states, january-may 2020

    MMWR. Morbidity and Mortality Weekly Report 69:795–800.

    https://doi.org/10.15585/mmwr.mm6925e2

    • PubMed
    • Google Scholar
27. 1. Mafham MM
    2. Spata E
    3. Goldacre R
    4. Gair D
    5. Curnow P
    6. Bray M
    7. Hollings S
    8. Roebuck C
    9. Gale CP
    10. Mamas MA
    11. Deanfield JE
    12. de Belder MA
    13. Luescher TF
    14. Denwood T
    15. Landray MJ
    16. Emberson JR
    17. Collins R
    18. Morris EJA
    19. Casadei B
    20. Baigent C

    (2020) COVID-19 pandemic and admission rates for and management of acute coronary syndromes in england

    Lancet 396:381–389.

    https://doi.org/10.1016/S0140-6736(20)31356-8

    • PubMed
    • Google Scholar
28. 1. Mena GE
    2. Martinez PP
    3. Mahmud AS
    4. Marquet PA
    5. Buckee CO
    6. Santillana M

    (2021) Socioeconomic status determines COVID-19 incidence and related mortality in santiago, chile

    Science 372:eabg5298.

    https://doi.org/10.1126/science.abg5298

    • PubMed
    • Google Scholar
29. 1. Meyerowitz-Katz G
    2. Merone L

    (2020) A systematic review and meta-analysis of published research data on COVID-19 infection fatality rates

    International Journal of Infectious Diseases 101:138–148.

    https://doi.org/10.1016/j.ijid.2020.09.1464

    • PubMed
    • Google Scholar
30. 1. Murray CJL
    2. Rajaratnam JK
    3. Marcus J
    4. Laakso T
    5. Lopez AD

    (2010) What can we conclude from death registration? improved methods for evaluating completeness

    PLOS Medicine 7:e1000262.

    https://doi.org/10.1371/journal.pmed.1000262

    • PubMed
    • Google Scholar
31. 1. Nørgaard SK
    2. Vestergaard LS
    3. Nielsen J
    4. Richter L
    5. Schmid D
    6. Bustos N
    7. Braye T
    8. Athanasiadou M
    9. Lytras T
    10. Denissov G
    11. Veideman T
    12. Luomala O
    13. Möttönen T
    14. Fouillet A
    15. Caserio-Schönemann C
    16. An der Heiden M
    17. Uphoff H
    18. Gkolfinopoulou K
    19. Bobvos J
    20. Paldy A
    21. Rotem N
    22. Kornilenko I
    23. Domegan L
    24. O’Donnell J
    25. Donato FD
    26. Scortichini M
    27. Hoffmann P
    28. Velez T
    29. England K
    30. Calleja N
    31. van Asten L
    32. Stoeldraijer L
    33. White RA
    34. Paulsen TH
    35. da Silva SP
    36. Rodrigues AP
    37. Klepac P
    38. Zaletel M
    39. Fafangel M
    40. Larrauri A
    41. León I
    42. Farah A
    43. Galanis I
    44. Junker C
    45. Perisa D
    46. Sinnathamby M
    47. Andrews N
    48. O’Doherty MG
    49. Irwin D
    50. Kennedy S
    51. McMenamin J
    52. Adlhoch C
    53. Bundle N
    54. Penttinen P
    55. Pukkila J
    56. Pebody R
    57. Krause TG
    58. Mølbak K

    (2021) Real-time monitoring shows substantial excess all-cause mortality during second wave of COVID-19 in europe, october to december 2020

    Euro Surveillance 26:2002023.

    https://doi.org/10.2807/1560-7917.ES.2021.26.1.2002023

    • PubMed
    • Google Scholar
32. 1. Odone A
    2. Delmonte D
    3. Gaetti G
    4. Signorelli C

    (2021) Doubled mortality rate during the COVID-19 pandemic in italy: quantifying what is not captured by surveillance

    Public Health 190:108–115.

    https://doi.org/10.1016/j.puhe.2020.11.016

    • PubMed
    • Google Scholar
33. Website

    1. Oxford University

    (2021) Covid-19 Government Response Tracker. USA state level Covid-19 Policy Responses

    Accessed December 1, 2021.

    https://github.com/OxCGRT/USA-covid-policy
34. 1. Prete CA
    2. Buss LF
    3. Buccheri R
    4. Abrahim CMM
    5. Salomon T
    6. Crispim MAE
    7. Oikawa MK
    8. Grebe E
    9. Fraiji NA
    10. Whittaker C
    11. Alexander N
    12. Faria NR
    13. Dye C
    14. Nascimento VH
    15. Busch MP
    16. Sabino EC

    (2022) Reinfection by the SARS-cov-2 gamma variant in blood donors in manaus, brazil

    BMC Infect Dis 22:127.

    https://doi.org/10.1186/s12879-022-07094-y

    • Google Scholar
35. 1. Quandelacy TM
    2. Viboud C
    3. Charu V
    4. Lipsitch M
    5. Goldstein E

    (2014) Age- and sex-related risk factors for influenza-associated mortality in the United States between 1997–2007

    American Journal of Epidemiology 179:156–167.

    https://doi.org/10.1093/aje/kwt235

    • PubMed
    • Google Scholar
36. 1. Reichert TA
    2. Simonsen L
    3. Sharma A
    4. Pardo SA
    5. Fedson DS
    6. Miller MA

    (2004) Influenza and the winter increase in mortality in the united states, 1959-1999

    American Journal of Epidemiology 160:492–502.

    https://doi.org/10.1093/aje/kwh227

    • PubMed
    • Google Scholar
37. 1. Rossen LM
    2. Ahmad FB
    3. Anderson RN
    4. Branum AM
    5. Du C
    6. Krumholz HM
    7. Li SX
    8. Lin Z
    9. Marshall A
    10. Sutton PD
    11. Faust JS

    (2021) Disparities in excess mortality associated with COVID-19 - united states, 2020

    MMWR. Morbidity and Mortality Weekly Report 70:1114–1119.

    https://doi.org/10.15585/mmwr.mm7033a2

    • PubMed
    • Google Scholar
38. Website

    1. Rudis B
    2. McGowan C
    3. Chen JJ
    4. Meyer S
    5. Turtle J
    6. Bates A
    7. McGovern I

    (2021) cdcfluview: Retrieve Flu Season Data from the United States Centers for Disease Control and Prevention ('CDC’) “FluView

    Portal (0.9.4. Accessed December 1, 2022.

    https://CRAN.R-project.org/package=cdcfluview
39. 1. Saha MV
    2. Davis RE
    3. Hondula DM

    (2014) Mortality displacement as a function of heat event strength in 7 us cities

    American Journal of Epidemiology 179:467–474.

    https://doi.org/10.1093/aje/kwt264

    • PubMed
    • Google Scholar
40. 1. Serfling RE

    (1963)

    Methods for current statistical analysis of excess pneumonia-influenza deaths

    Public Health Reports 78:494–506.

    • PubMed
    • Google Scholar
41. 1. Sharma R
    2. Kuohn LR
    3. Weinberger DM
    4. Warren JL
    5. Sansing LH
    6. Jasne A
    7. Falcone G
    8. Dhand A
    9. Sheth KN

    (2021) Excess cerebrovascular mortality in the united states during the COVID-19 pandemic

    Stroke 52:563–572.

    https://doi.org/10.1161/STROKEAHA.120.031975

    • PubMed
    • Google Scholar
42. 1. van Asten L
    2. Harmsen CN
    3. Stoeldraijer L
    4. Klinkenberg D
    5. Teirlinck AC
    6. de Lange MMA
    7. Meijer A
    8. van de Kassteele J
    9. van Gageldonk-Lafeber AB
    10. van den Hof S
    11. van der Hoek W

    (2021) Excess deaths during influenza and coronavirus disease and infection-fatality rate for severe acute respiratory syndrome coronavirus 2, the netherlands

    Emerging Infectious Diseases 27:411–420.

    https://doi.org/10.3201/eid2702.202999

    • PubMed
    • Google Scholar
43. 1. Verity R
    2. Okell LC
    3. Dorigatti I
    4. Winskill P
    5. Whittaker C
    6. Imai N
    7. Cuomo-Dannenburg G
    8. Thompson H
    9. Walker PGT
    10. Fu H
    11. Dighe A
    12. Griffin JT
    13. Baguelin M
    14. Bhatia S
    15. Boonyasiri A
    16. Cori A
    17. Cucunubá Z
    18. FitzJohn R
    19. Gaythorpe K
    20. Green W
    21. Hamlet A
    22. Hinsley W
    23. Laydon D
    24. Nedjati-Gilani G
    25. Riley S
    26. van Elsland S
    27. Volz E
    28. Wang H
    29. Wang Y
    30. Xi X
    31. Donnelly CA
    32. Ghani AC
    33. Ferguson NM

    (2020) Estimates of the severity of coronavirus disease 2019: a model-based analysis

    The Lancet. Infectious Diseases 20:669–677.

    https://doi.org/10.1016/S1473-3099(20)30243-7

    • PubMed
    • Google Scholar
44. Software

    1. Viboud C

    (2023) DirectIndirectCOVID19MortalityEstimation, version swh:1:rev:f0c23c87a00f589179af5351e89732f51d5a2b19

    Software Heritage.

    https://archive.softwareheritage.org/swh:1:dir:1fa09d8b19c99ca1fa8920eacef451bfa76f2e75;origin=https://github.com/viboudc/DirectIndirectCOVID19MortalityEstimation;visit=swh:1:snp:720b8da3a3f85c5faa65b77f8cada7dcce1833d3;anchor=swh:1:rev:f0c23c87a00f589179af5351e89732f51d5a2b19
45. 1. Weinberger DM
    2. Chen J
    3. Cohen T
    4. Crawford FW
    5. Mostashari F
    6. Olson D
    7. Pitzer VE
    8. Reich NG
    9. Russi M
    10. Simonsen L
    11. Watkins A
    12. Viboud C

    (2020) Estimation of excess deaths associated with the COVID-19 pandemic in the united states, march to may 2020

    JAMA Internal Medicine 180:1336–1344.

    https://doi.org/10.1001/jamainternmed.2020.3391

    • PubMed
    • Google Scholar
46. 1. Woolf SH
    2. Chapman DA
    3. Sabo RT
    4. Weinberger DM
    5. Hill L

    (2020) Excess deaths from COVID-19 and other causes, march-april 2020

    JAMA 324:510–513.

    https://doi.org/10.1001/jama.2020.11787

    • PubMed
    • Google Scholar
47. 1. Woolf SH
    2. Chapman DA
    3. Sabo RT
    4. Zimmerman EB

    (2021) Excess deaths from COVID-19 and other causes in the US

    JAMA 325:1786–1789.

    https://doi.org/10.1001/jama.2021.5199

    • Google Scholar

Decision letter after peer review:

Thank you for submitting your article "Direct and indirect mortality impacts of the COVID-19 pandemic in the US, March 2020-April 2021" for consideration by eLife. Your article has been reviewed by 3 peer reviewers, one of whom is a member of our Board of Reviewing Editors, and the evaluation has been overseen by David Serwadda as the Senior Editor. The reviewers have opted to remain anonymous.

The reviewers have discussed their reviews with one another, and the Reviewing Editor has drafted this to help you prepare a revised submission.

Essential revisions:

This is a very interesting paper quantifying excess deaths due to the COVID-19 pandemic in the USA. The paper is roughly divided into two main sections. In the first section, the authors estimate age and cause-specific excess mortality. In the second section, using their excess mortality estimates, the authors attempt to disentangle the impact of SARS-CoV-2 infection (direct impact) vs. the impact of NPIs on this excess mortality (indirect impact). While all reviewers agree that this is a relevant contribution, the paper needs some revisions/clarifications, particularly in the second section.

1. We would encourage authors to discuss the two different concepts of excess mortality:

(#1) what deaths were caused, directly or indirectly, by the pandemic. This is what authors seem to have aimed to assess (#2) how many additional deaths occurred during the pandemic, compared to what would have been expected in the absence of a pandemic. For such an analysis I think expected annual influenza deaths should be added back to the baseline (or subtracted from the excess)? Some of the discussion seems to relate more to an impression of #2 rather than #1 so it would be good to be explicit about what is being estimated.

2. Choosing the baseline/counterfactual is a key part of excess mortality analyses. I've seen other papers exploring different potential periods (shorter periods; either by using older or newer years, e,g 2017-2020 or 2014-2017) to test sensitivity to the choice of these periods. Moreover, I would think that an absence of other point-in-time disasters would also be a needed assumption (as these would not represent a baseline). It may be beneficial for the reader to outline explicitly the assumptions of their model. Some of these assumptions are outlined on page 5, but I am not sure whether this is an extensive list of the assumptions of this approach.

3. There are no mentions in the description of mortality data of two common issues with vital registration data. First, the presence of ill-defined deaths (R chapter, and when subdividing injuries, Y10-Y34). If these increase or decrease during the pandemic because of differentials in coding, it may over or underestimate the presence of excess mortality. Second, the lack of coverage of all deaths. Helleringer talks a bit about that here https://academic.oup.com/ije/article/51/1/85/6460626?login=true in the context of estimating excess mortality. While this issue may seem trivial in the US, Murray shows here that there is a lack of complete coverage that is differential by county in the US https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1000262. While this may or may not affect the authors' analysis, I think the implicit assumption of no changes in the coverage of deaths should be made explicit.

4. We are concerned about the analysis estimating the impact of NPIs. The authors are using explicit causal language throughout this section, but there's very little discussion about the methods used and the assumptions and limitations of these methods. Specifically:

a. Please provide a bit more information on the model used for direct vs. indirect effects. We'd like to see explicit discussion of the assumptions and limitations of the approach but also of the stringency index used. Was the association between stringency index and excess deaths assumed to be linear? Or were different functional forms considered? It is also not clear how well the model fit the data.

b. Related to the above, please provide more details on how the results of the regressions were translated into the results presented. The main text reports percentages, but the methods only briefly explain how numbers of direct deaths were calculated, and the supplementary tables report coefficients. It is not clear if these estimates of direct and indirect deaths were somehow constrained to add up to the total number of excess deaths, but it doesn't seem like it since point estimates cross 100% in some cases.

c. Please be more explicit about causal assumptions (see comment 6 of reviewer 2).

d. Please discuss the potential limitations of using the stringency index to quantify NPIs particularly since it doesn't take into account the actual implementation of the measures (and how this may have varied over time).

Reviewer #2 (Recommendations for the authors):

In this paper, the authors examine the impacts of the COVID-19 pandemic on excess mortality in the US up to April 30, 2021. The authors separate direct impacts (caused by COVID-19, coded as such or not) of the pandemic from indirect impacts (disruptions), finding that most excess deaths (90%) are due to direct impacts. Importantly, the authors find that the official COVID-19 death tally is an undercount of these deaths. Moreover, the authors also find that excess deaths due to other causes are the main driver of excess mortality among younger populations. This is a very important research question that the authors have addressed using robust methods. I have a few suggestions that I hope can help clarify the reporting of results and outlining of assumptions.

1. Abstract: since the study presents several axes of comparison (old vs young, direct vs indirect, specific causes vs one another) the abstract is quite complicated, and there are a few sections that may benefit from clarification. Specifically: (a) the authors mention the indirect effects

2. The framing of the study in the introduction may benefit from a more comprehensive outlining of potential directions that mortality may change with the pandemic. For example, declines in mortality have been observed in many settings (e.g. Chile and Guatemala; or in Figure 3 of this paper) following lockdowns in areas with low COVID-19 spread. This is mostly due to declines in external causes (mostly traffic crashes) and air pollution-related deaths (e.g. CVD or infant deaths). The authors just mention one potential directionality (COVID-19 increasing some deaths directly, and NPIs causing other deaths) without any mention to a lowering of mortality that may obscure these increases.

3. Choosing the baseline/counterfactual is a key part of excess mortality analyses. I've seen other papers exploring different potential periods (shorter periods; either by using older or newer years, e,g 2017-2020 or 2014-2017) to test sensitivity to the choice of these periods. Moreover, I would think that an absence of other point-in-time disasters would also be a needed assumption (as these would not represent a baseline). It may be beneficial for the reader to outline explicitly the assumptions of their model. Some of these assumptions are outlined on page 5, but I am not sure whether this is an extensive list of the assumptions of this approach.

4. I see no mentions in the description of mortality data of two common issues with vital registration data. First, the presence of ill-defined deaths (R chapter, and when subdividing injuries, Y10-Y34). If these increase or decrease during the pandemic because of differentials in coding, it may over or underestimate the presence of excess mortality. Second, the lack of coverage of all deaths. Helleringer talks a bit about that here https://academic.oup.com/ije/article/51/1/85/6460626?login=true in the context of estimating excess mortality. While this issue may seem trivial in the US, Murray shows here that there is a lack of complete coverage that is differential by county in the US https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1000262. While this may or may not affect the authors' analysis, I think the implicit assumption of no changes in the coverage of deaths should be made explicit.

5. Direct deaths: the authors use COVID-19, influenza, pneumonia, and COPD deaths as an indicator of respiratory mortality. Is this the list of ICD codes in P28 (2.a) of the appendix? The authors also use "mention of COVID-19" as an indicator of directly caused deaths. How is this operationalized? Is it ICD10 code U07? Or does this not rely on ICD coding? The list of respiratory codes in the supplement makes it seem like this is ICD based, but it is unclear from the text itself.

6. I am concerned about the analysis estimating the impact of NPIs. The authors are using explicit causal language throughout this section, but there's very little discussion about the causal assumptions underpinning this analysis. Specifically: (a) I see no approach to addressing lack of exchangeability (for example, areas that implemented more NPIs may have a more vulnerable population, hence the political will to implement them); (b) no discussion of issues with levels of analysis from a jurisdictional point of view, since the level of devolvement of power from states to local jurisdictions varies widely by state, making state-level analysis potentially tricky (and county-level ones too for that matter); and (c) issues with measurement error, specifically differential measurement error of the outcome, influenced by the exposure (improved vital registration coding in areas with more or less NPIs due to better established public health systems). Last, the authors are framing this analysis as a way to "quantify the direct and indirect impacts of the pandemic on different causes of death". I am failing to grasp the intuition of how this analysis helps with this objective, so it may need to be explicitly set in the methods section.

7. I want to commend the authors for their presentation of results. Specifically, the rounding of mortality estimates to avoid false notions of precision and very clear figures. A few notes on figures: (a) figure 1 is lacking a title in my PDF, and the black line is not included in the legend I am guessing it corresponds to the state in the facet title; (b) figure 2 is missing confidence intervals for respiratory illnesses; (c) I suggest the authors map the results of table 1, as a spatial pattern will most likely emerge (top by excess mortality are all Deep South or Northeast states).

8. The results comparing COVID-19 to the 2017/2018 flu season are very interesting, as they help contextualize the impact. However, I am a bit concerned about "denominators" here, specifically time. The authors compared 13 months of excess mortality to an undetermined number of months (4? 6? 12?). I am not sure how influenza seasons are conceptualized, and I think it'd be good to be explicit with its duration. I understand that the authors may be wanting to compare a "full season" of influenza vs one of COVID-19, but is this direct comparison even possible, considering the different (or lack of?) seasonality of COVID-19?

9. There's another result I am having trouble (a) interpreting it and (b) understanding where it comes from: % of deaths directly attributable to COVID-19. The authors seem to be interpreting this as direct effects of COVID-19 infection, as opposed to indirect effects due to disruptions, at least according to the appendix. My comment 2 above already covers the idea that only thinking about increases in mortality due to NPIs is an incomplete framing, as there are reasons why we expect deaths to actually decline during lockdowns (traffic or air pollution-related ones). There's a cursory mention of this in the appendix, but it is restricted to other respiratory deaths. To follow the authors' example in the appendix, a person with diabetes may also not have died because the air pollution levels in their city plummeted and the MI they were about to have did not happen. Or because their car didn't crash (or another car didn't crash on them) because traffic levels plummeted too. On (b), understanding where it comes from, the appendix provides extra clues (the main text doesn't explain this part that much). If I am understanding things right, the authors are regressing age and cause-specific mortality on NPI strength and on weekly deaths. They are then multiplying the COVID-19 deaths coefficient by the number of deaths to get the # of deaths directly caused by COVID-19. The authors do test potential lags, as both NPIs and deaths may have lagged effects or may be the result of lags between infections and deaths. However, I don't fully follow: (a) why the NPI adjustment is needed at all, (b) why are deaths an adequate COVID-19 activity indicator, especially since authors find a 0-week lag (pretty self-fulfilling, given that they are regressing deaths on a subset of deaths), which doesn't speak much about healthcare disruptions (which occur weeks before that point). One last concern I have is of competing risks, and whether areas with more strict NPIs would have lower direct excess mortality, increasing the % of deaths that are indirectly attributable to the pandemic, not because these deaths were caused by the pandemic, but because they could not have been caused by a well-controlled epidemic (given NPIs).

Essential revisions:

This is a very interesting paper quantifying excess deaths due to the COVID-19 pandemic in the USA. The paper is roughly divided into two main sections. In the first section, the authors estimate age and cause-specific excess mortality. In the second section, using their excess mortality estimates, the authors attempt to disentangle the impact of SARS-CoV-2 infection (direct impact) vs. the impact of NPIs on this excess mortality (indirect impact). While all reviewers agree that this is a relevant contribution, the paper needs some revisions/clarifications, particularly in the second section.

We thank the editor and reviewers for their constructive comments. We acknowledge that the second section needed some clarifications and improvement with respect to the hypotheses tested. We have revised the paper accordingly and added more detailed explanations to the introduction, methods and supplement. We have also updated our models (second section) to include non-linear effects between excess mortality and covariates and included the potential effect of increased hospital occupancy use during COVID-19. Further, we have extended the analysis until January 1, 2022, adding 8 months of data (which is effectively adding the Δ variant wave to the initial analysis). We ran sensitivity analyses on the length of the data used for calibration of the model baseline. We have also revised the comparison between COVID19 and influenza. As a result, the paper has been substantially rewritten and overhauled. These changes have strengthened our analyses, but they do not have any bearing on our conclusions.

1. We would encourage authors to discuss the two different concepts of excess mortality:

(#1) what deaths were caused, directly or indirectly, by the pandemic. This is what authors seem to have aimed to assess (#2) how many additional deaths occurred during the pandemic, compared to what would have been expected in the absence of a pandemic. For such an analysis I think expected annual influenza deaths should be added back to the baseline (or subtracted from the excess)? Some of the discussion seems to relate more to an impression of #2 rather than #1 so it would be good to be explicit about what is being estimated.

Thanks for the comment. We have clarified our assumptions and justification for the construction of the baseline. Overall, our excess mortality approach estimates deaths above mortality from past years; here we single out influenza by adding a covariate for flu in the regression and take it out of the baseline. Excess deaths due to COVID19 are deaths above the baseline estimated from prior years, after the impact of influenza has been removed. Our choice is motivated by the large variation in influenza mortality burden between years due to differences in circulating strains – it is not straightforward to define an average flu season. On the other hand, we do not single out deaths from many other viral infections that are presumably less variable between years (for which we do not have viral activity proxies) – these viruses have been greatly reduced during the pandemic and hence this means our baseline is likely inflated.

We have now clarified our assumptions about the baseline in the methods section (a slightly condensed version of the above response is provided in the first paragraph of analytic approach, p 5). Also, we return to the meaning of the baseline in discussion (limitations section, bottom of p12). Further, we now provide a comparison of influenza and COVID-19 excess mortality for a comparable period of virus circulation (bottom of p7).

2. Choosing the baseline/counterfactual is a key part of excess mortality analyses. I've seen other papers exploring different potential periods (shorter periods; either by using older or newer years, e,g 2017-2020 or 2014-2017) to test sensitivity to the choice of these periods. Moreover, I would think that an absence of other point-in-time disasters would also be a needed assumption (as these would not represent a baseline). It may be beneficial for the reader to outline explicitly the assumptions of their model. Some of these assumptions are outlined on page 5, but I am not sure whether this is an extensive list of the assumptions of this approach.

These are fair points. We do not know of any other point-in-time disasters that would have occurred in our study period but cannot entirely rule out such contribution. We have added a caveat to the list of limitations at the top of p13.

To address the issue of sensitivity to the number of years included in the baseline, we have explored a shorter historical window of 4 years (2016-2020) instead of 6 years in the original analysis (2014-2020). The results are shown in Figure S9 and commented in the Results section (top of p 7) and discussion (limitations, top of p 13). Estimates of all-cause and respiratory excess mortality were extremely robust to the length of the calibration data for the model baseline, and so were all national cause-specific estimates. State- and cause-specific estimates were more sensitive to this choice, especially for Alzheimer’s and heart diseases, due to short-term trends and small counts in some of the time series. We note that our main conclusions are based on national estimates, which were less sensitive to the choice of calibration data.

3. There are no mentions in the description of mortality data of two common issues with vital registration data. First, the presence of ill-defined deaths (R chapter, and when subdividing injuries, Y10-Y34). If these increase or decrease during the pandemic because of differentials in coding, it may over or underestimate the presence of excess mortality. Second, the lack of coverage of all deaths. Helleringer talks a bit about that here https://academic.oup.com/ije/article/51/1/85/6460626?login=true in the context of estimating excess mortality. While this issue may seem trivial in the US, Murray shows here that there is a lack of complete coverage that is differential by county in the US https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1000262. While this may or may not affect the authors' analysis, I think the implicit assumption of no changes in the coverage of deaths should be made explicit.

We added a mention of coverage issues and changes in reporting of ill-defined deaths to the limitation section (top of p13) and refer to the Murray paper. We note that ill-defined deaths would be captured in all-cause mortality but not in cause-specific mortality. Further, codes Y10-34 (unintentional intent injuries) are combined with the codes reflecting known intent injuries in the mortality datasets that we use here, so a large change in these codes would likely be captured in our analyses.

4. We are concerned about the analysis estimating the impact of NPIs. The authors are using explicit causal language throughout this section, but there's very little discussion about the methods used and the assumptions and limitations of these methods. Specifically:

a. Please provide a bit more information on the model used for direct vs. indirect effects. We'd like to see explicit discussion of the assumptions and limitations of the approach but also of the stringency index used. Was the association between stringency index and excess deaths assumed to be linear? Or were different functional forms considered? It is also not clear how well the model fit the data.

Thank you. We have entirely revamped our analysis of direct and indirect effects and provided more information in the main text and supplement. A list of these changes is provided here:

1. We now use generalized additive models (gam) which allow for non-linear relationships between covariates and excess mortality. Indeed, we do find evidence for non-linear relationships with covariates, particularly with NPI.

2. Our initial analysis was focused on predictors measuring COVID19 activity and the Oxford index, which is a broad indicator of the sum of NPIs in any community. We have now expanded the analysis to include hospital ICU occupancy in the model, as there could be a separate effect from COVID19 intensity and NPIs.

3. We have also provided model fit statistics in the text and several figures to illustrate model projections vs observations (Figures 4 and S16-19).

4. We have added caveats to the discussion p 13: “Though gam models indicate that the strength of interventions is a dominant predictor of excess mortality from external causes and all-cause mortality in young age groups, the relationships are non-linear and the resulting models do not fully capture mortality changes during the pandemic.”

5. We have combed through the paper to remove the causal language and emphasized that the statistical relationships with interventions are not causal and further exploration of the mechanisms at play is needed.

b. Related to the above, please provide more details on how the results of the regressions were translated into the results presented. The main text reports percentages, but the methods only briefly explain how numbers of direct deaths were calculated, and the supplementary tables report coefficients. It is not clear if these estimates of direct and indirect deaths were somehow constrained to add up to the total number of excess deaths, but it doesn't seem like it since point estimates cross 100% in some cases.

That’s correct; estimates are not constrained to 100%. We have provided more details about the estimation of the percentages (=attributable fraction) in the main text (middle of p6) and in the supplement (p 4-5). Briefly, to estimate the direct impact of COVID19, we calculate expected excess deaths with the full gam model, where all predictors are set to their observed values, and with a zero-COVID19 model, where the COVID19 predictor is set to zero. The estimated fraction attributable to COVID19 (direct impact of the pandemic) is the difference between the predicted values by the full model and the zero-COVID19 model, divided by the predictions of the full model. The full equation is provided in supplement p4-5. Confidence intervals on these percentages are based on resampling excess mortality estimates, refitting the gam models, and repeating the above procedure 1000 times for each mortality stratum.

We think important to not restrict estimates to remain below 100% since we find consistent percentages above 100% in both gam approach and the “empirical” comparison with official statistics (Table 4, estimates for seniors). These results point at the possible effects of harvesting and/or inflated baseline due to depressed pathogen circulation.

c. Please be more explicit about causal assumptions (see comment 6 of reviewer 2).

We have clarified the attribution model in methods and supplement and removed causal language.

d. Please discuss the potential limitations of using the stringency index to quantify NPIs particularly since it doesn't take into account the actual implementation of the measures (and how this may have varied over time).

We have added several sentences to this effect in the caveats section p 13. “Generalized additive models indicate that a proxy for the strength of interventions is a dominant predictor of excess mortality from external causes and all-cause mortality in individuals under 45 yrs. Yet, the relationships are non-linear, and the resulting models do not fully capture mortality changes during the pandemic. Along the same lines, the Oxford stringency index used as a proxy for interventions does not consider the actual implementation nor the effect of interventions; it is solely based on mandates in place in different locations and time periods. We also assume that, for a given level of stringency, the impact of interventions does not change over time. Because time and intervention stringency are highly conflated, it would be difficult to study temporal variation in this relationship.”

Reviewer #2 (Recommendations for the authors):

In this paper, the authors examine the impacts of the COVID-19 pandemic on excess mortality in the US up to April 30, 2021. The authors separate direct impacts (caused by COVID-19, coded as such or not) of the pandemic from indirect impacts (disruptions), finding that most excess deaths (90%) are due to direct impacts. Importantly, the authors find that the official COVID-19 death tally is an undercount of these deaths. Moreover, the authors also find that excess deaths due to other causes are the main driver of excess mortality among younger populations. This is a very important research question that the authors have addressed using robust methods. I have a few suggestions that I hope can help clarify the reporting of results and outlining of assumptions.

1. Abstract: since the study presents several axes of comparison (old vs young, direct vs indirect, specific causes vs one another) the abstract is quite complicated, and there are a few sections that may benefit from clarification. Specifically: (a) the authors mention the indirect effects

It looks like the end of this comment may have been cut; but the point about the abstract being complicated is well taken. We have hopefully simplified the abstract.

2. The framing of the study in the introduction may benefit from a more comprehensive outlining of potential directions that mortality may change with the pandemic. For example, declines in mortality have been observed in many settings (e.g. Chile and Guatemala; or in Figure 3 of this paper) following lockdowns in areas with low COVID-19 spread. This is mostly due to declines in external causes (mostly traffic crashes) and air pollution-related deaths (e.g. CVD or infant deaths). The authors just mention one potential directionality (COVID-19 increasing some deaths directly, and NPIs causing other deaths) without any mention to a lowering of mortality that may obscure these increases.

Thanks. We do mention the potential mortality declines brought about by lockdown in discussion (end of p13). In the revised introduction (p3), we have provided a fuller list of mechanisms that may increase or decrease mortality during the pandemic and explained how we would expect to see these mechanisms manifest in data.

3. Choosing the baseline/counterfactual is a key part of excess mortality analyses. I've seen other papers exploring different potential periods (shorter periods; either by using older or newer years, e,g 2017-2020 or 2014-2017) to test sensitivity to the choice of these periods. Moreover, I would think that an absence of other point-in-time disasters would also be a needed assumption (as these would not represent a baseline). It may be beneficial for the reader to outline explicitly the assumptions of their model. Some of these assumptions are outlined on page 5, but I am not sure whether this is an extensive list of the assumptions of this approach.

The reviewer makes a good point that the choice of historic years used for calibration of the model baseline is important. We do not know of any other point-in-time disasters that would have occurred in our study period.

To address the issue of sensitivity to the number of years included in baseline, we have tried a shorter historical window of ~4 years (2016-2020) instead of ~6 years used in the original analysis (2014-2020). The results are shown in Figure S9 and commented in the Results section (top of p 7) and discussion (limitations, bottom of p 12). Estimates of all-cause and respiratory excess mortality were extremely robust to the length of the calibration data used for the model baseline, as well as national estimates for any cause. State- and cause-specific estimates were more sensitive to this choice, especially for Alzheimer’s and heart diseases, due to short-term trends and small counts in some of the time series. We note that our main conclusions are based on national estimates, which were less sensitive to the choice of calibration data.

4. I see no mentions in the description of mortality data of two common issues with vital registration data. First, the presence of ill-defined deaths (R chapter, and when subdividing injuries, Y10-Y34). If these increase or decrease during the pandemic because of differentials in coding, it may over or underestimate the presence of excess mortality. Second, the lack of coverage of all deaths. Helleringer talks a bit about that here https://academic.oup.com/ije/article/51/1/85/6460626?login=true in the context of estimating excess mortality. While this issue may seem trivial in the US, Murray shows here that there is a lack of complete coverage that is differential by county in the US https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1000262. While this may or may not affect the authors' analysis, I think the implicit assumption of no changes in the coverage of deaths should be made explicit.

We added a mention of coverage issues and changes in reporting of ill-defined deaths to the limitation section and refer to the Murray paper. We note that ill-defined deaths would be captured in all-cause mortality but not in cause-specific mortality. Further, codes Y10-34 (unintentional intent injuries) are combined with the codes reflecting known intent injuries in the real time mortality data, so a large change in these codes would probably be captured in our analyses.

5. Direct deaths: the authors use COVID-19, influenza, pneumonia, and COPD deaths as an indicator of respiratory mortality. Is this the list of ICD codes in P28 (2.a) of the appendix? The authors also use "mention of COVID-19" as an indicator of directly caused deaths. How is this operationalized? Is it ICD10 code U07? Or does this not rely on ICD coding? The list of respiratory codes in the supplement makes it seem like this is ICD based, but it is unclear from the text itself.

The list of ICD codes is provided in supplement p1. Our entire analysis is based on ICD codes, and we use U07 for COVID-19. We have clarified this in methods of the main text (p4). Note that throughout the paper, we use the aggregated time series made available in near real time throughout the pandemic by CDC/NCHS, so the data have already been tabulated by select categories of codes and we cannot create different categories.

6. I am concerned about the analysis estimating the impact of NPIs. The authors are using explicit causal language throughout this section, but there's very little discussion about the causal assumptions underpinning this analysis. Specifically: (a) I see no approach to addressing lack of exchangeability (for example, areas that implemented more NPIs may have a more vulnerable population, hence the political will to implement them); (b) no discussion of issues with levels of analysis from a jurisdictional point of view, since the level of devolvement of power from states to local jurisdictions varies widely by state, making state-level analysis potentially tricky (and county-level ones too for that matter); and (c) issues with measurement error, specifically differential measurement error of the outcome, influenced by the exposure (improved vital registration coding in areas with more or less NPIs due to better established public health systems). Last, the authors are framing this analysis as a way to "quantify the direct and indirect impacts of the pandemic on different causes of death". I am failing to grasp the intuition of how this analysis helps with this objective, so it may need to be explicitly set in the methods section.

We have expanded our list of caveats for this analysis based on the comments by this and other reviewers (p 13, the full section is also appended below). In particular, we have added comments regarding differential vulnerability of the populations, changes in coding and coverage of vital statistics during COVID-19, and lack of consideration of NPI implementation at the local level.

Limitation section p 13: “Our study is subject to several limitations. First, mortality counts below the minimum cut-off value of 10 were suppressed due to privacy regulations. As a result, our age-specific analyses were restricted to larger states, and we could not assess the role of race/ethnicity. Prior work has shown important disparities in COVID-19 impact by race/ethnicity and economic status (Mena et al., 2021; Rossen, 2021) in the US and abroad. Second, official coding practices may have changed between states and through time based on SARS-CoV-2 testing availability, location of death, demographic factors, and comorbidities. Third, we assumed full coverage of deaths reporting, which may not be valid throughout the US (Murray et al., 2010), and we did not study changes in deaths ascribed to ill-defined codes (R codes). Ill-defined deaths would be captured in all-cause mortality but not in cause-specific analyses. Fourth, we find periods of negative excesses in cancer (throughout the pandemic), cardiovascular, and heart diseases, possibly due to changes in ascertainment of underlying cause of death (e.g. a death in a cancer patient with COVID-19 is ascribed to COVID-19), harvesting (Saha et al., 2013), or depressed circulation of endemic pathogens other than influenza. Fifth, we choose to fit the model baseline to data for 2014-2020, which is arbitrary. We studied the sensitivity of our excess mortality estimates to this assumption (Supplement). While national analyses were robust to this choice, as well as state-level analyses of most conditions, state-specific estimates for Alzheimer’s disease were more sensitive. Further, the model baseline does not account for possible point-in-time disasters that may have occurred during the pandemic but are independent of COVID19 (eg, a hurricane) or changes in air pollution. Sixth, generalized additive models indicate that the Oxford stringency index, a proxy for the strength of interventions, is a dominant predictor of excess mortality from external causes and all-cause mortality in individuals under 45 yrs. Yet, the relationships are non-linear, and the resulting models do not fully capture mortality changes during the pandemic. Along the same lines, the Oxford stringency index does not consider the actual implementation nor the effect of interventions; it is solely based on mandates in place in different locations and time periods. We also assume that, for a given level of stringency, the impact of interventions does not change over time. Because time and intervention stringency are highly conflated, it would be difficult to study potential temporal variation in this relationship. Further, analyses are aggregated at the state or national level, while implementation of interventions may operate more locally. We also do not account for underlying differences in vulnerability between states, where more vulnerable states may have implemented stricter interventions (although this potential bias would not affect temporal analyses). Finally, our study ends on January 1, 2022 and does not capture a recrudescence of COVID-19-related deaths due to the Omicron variant. As a result, our excess mortality estimates should be deemed conservative.”

7. I want to commend the authors for their presentation of results. Specifically, the rounding of mortality estimates to avoid false notions of precision and very clear figures. A few notes on figures: (a) figure 1 is lacking a title in my PDF, and the black line is not included in the legend (I am guessing it corresponds to the state in the facet title); (b) figure 2 is missing confidence intervals for respiratory illnesses; (c) I suggest the authors map the results of table 1, as a spatial pattern will most likely emerge (top by excess mortality are all Deep South or Northeast states).

Thank you. We have added a title to figure 1 and a legend for the back line. As regards mapping all-cause excess mortality, now that every state has experienced multiple waves of COVID19, and each state was hit hard at a different time, the geography of excess mortality for the combined study period of March 1, 2020 to January 1, 2022 is not as meaningful.

8. The results comparing COVID-19 to the 2017/2018 flu season are very interesting, as they help contextualize the impact. However, I am a bit concerned about "denominators" here, specifically time. The authors compared 13 months of excess mortality to an undetermined number of months (4? 6? 12?). I am not sure how influenza seasons are conceptualized, and I think it'd be good to be explicit with its duration. I understand that the authors may be wanting to compare a "full season" of influenza vs one of COVID-19, but is this direct comparison even possible, considering the different (or lack of?) seasonality of COVID-19?

The reviewer is correct that these comparisons were for different epidemic durations. We have revamped this section. It is tricky to find a particular time period that is relevant for comparison since COVID19 mortality has been quite heterogeneous across the US. Yet we elected to use the November to March period for both pathogens, comparing the severe A/H3N2 flu season in Nov 2017-Mar 2018, with the first winter wave of COVID19 from Nov 2020-Mar 2021 (where excess mortality was experienced by all states). This comparison highlights the magnitude of COVID-19, where excess mortality is 5-6 fold higher than excess mortality from flu, just for these 5 months. The relevant sections of results (middle of p 7) and Appendix have been updated and a new figure was added to supplement (Figure S11).

9. There's another result I am having trouble (a) interpreting it and (b) understanding where it comes from: % of deaths directly attributable to COVID-19. The authors seem to be interpreting this as direct effects of COVID-19 infection, as opposed to indirect effects due to disruptions, at least according to the appendix. My comment 2 above already covers the idea that only thinking about increases in mortality due to NPIs is an incomplete framing, as there are reasons why we expect deaths to actually decline during lockdowns (traffic or air pollution-related ones). There's a cursory mention of this in the appendix, but it is restricted to other respiratory deaths. To follow the authors' example in the appendix, a person with diabetes may also not have died because the air pollution levels in their city plummeted and the MI they were about to have did not happen. Or because their car didn't crash (or another car didn't crash on them) because traffic levels plummeted too. On (b), understanding where it comes from, the appendix provides extra clues (the main text doesn't explain this part that much). If I am understanding things right, the authors are regressing age and cause-specific mortality on NPI strength and on weekly deaths. They are then multiplying the COVID-19 deaths coefficient by the number of deaths to get the # of deaths directly caused by COVID-19. The authors do test potential lags, as both NPIs and deaths may have lagged effects or may be the result of lags between infections and deaths. However, I don't fully follow: (a) why the NPI adjustment is needed at all, (b) why are deaths an adequate COVID-19 activity indicator, especially since authors find a 0-week lag (pretty self-fulfilling, given that they are regressing deaths on a subset of deaths), which doesn't speak much about healthcare disruptions (which occur weeks before that point).

We do not consider NPI as an adjustment, merely as a potential covariate representing a broad set of interventions that may affect mortality. As the reviewer points out, an effect of interventions on mortality could be either positive or negative, for instance lockdown reducing pollution in turn reducing cardiac deaths, or lockdown increasing mental health issues in turn increasing suicide or opioids deaths. We have now included a fuller list of mechanisms involving interventions in the introduction (including air pollution) to clarify our thinking and justify the choice of the model with COVID indicator and NPI (p3 of introduction). As regards healthcare disruptions, we have expanded the attribution model to include ICU occupancy.

Our approach for estimating the % of deaths directly due to COVID19 has changed as well. We use a gam model to identify the relationship between excess deaths and covariates (COVID19, NPI and ICU), and compare the excess mortality predictions of the full gam model with COVID19, NPI and ICU covariates to their observed values, vs those of the same model but where COVID19 is set to zero. This is now detailed in methods and supplement.

One last concern I have is of competing risks, and whether areas with more strict NPIs would have lower direct excess mortality, increasing the % of deaths that are indirectly attributable to the pandemic, not because these deaths were caused by the pandemic, but because they could not have been caused by a well-controlled epidemic (given NPIs).

We try to address this point by discussing countries which have experienced lockdowns without a COVID19 epidemic (eg Australia before Omicron, Russia before the wild-type variant arrived, end p 13). In essence, if NPIs work to control SARS circulation, then mortality should either be at baseline levels (hence excess mortality is zero), or below baseline levels (negative excess mortality). Australia and Russia seem to have experienced the former (zero excess mortality).

Author details

1. Wha-Eum Lee

   Department of Ecology and Evolutionary Biology, Princeton University, Princeton, United States

   Contribution

   Data curation, Formal analysis, Visualization, Methodology, Writing – original draft, Writing – review and editing

   Competing interests

   No competing interests declared

2. Sang Woo Park

   Department of Ecology and Evolutionary Biology, Princeton University, Princeton, United States

   Contribution

   Supervision, Methodology, Writing – review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-2202-3361

3. Daniel M Weinberger

   School of Public Health, Yale University, New Haven, United States

   Contribution

   Conceptualization, Software, Methodology, Writing – review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-1178-8086

4. Donald Olson

   New York City Department of Health and Mental Hygiene, New York, United States

   Contribution

   Conceptualization, Methodology, Writing – review and editing

   Competing interests

   No competing interests declared

5. Lone Simonsen

   Department of Science and Environment, Roskilde University, Roskilde, Denmark

   Contribution

   Conceptualization, Writing – review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-1535-8526

6. Bryan T Grenfell

   1. Department of Ecology and Evolutionary Biology, Princeton University, Princeton, United States
   2. Princeton School of Public Affairs, Princeton University, Princeton, United States

   Contribution

   Conceptualization, Supervision, Writing – review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-3227-5909

7. Cécile Viboud

   Division of International Epidemiology and Population Studies, Fogarty International Center, National Institutes of Health, Bethesda, United States

   Contribution

   Conceptualization, Data curation, Formal analysis, Visualization, Methodology, Writing – original draft, Writing – review and editing

   For correspondence

   viboudc@mail.nih.gov

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-3243-4711

• 1,995

  Page views

• 168

  Downloads

• 5

  Citations

Article citation count generated by polling the highest count across the following sources: Crossref, PubMed Central, Scopus.