Data recently reported by Hoppe et al., 2019 from the National Institutes of Health ('NIH') suggest that part of the well-documented funding disparity (Ginther et al., 2011) affecting African-American Black ('AAB') principal investigators ('PIs') may be related to the topic of their applications. The authors of that report (including the first author of this report) found that topic choice accounted for over 20% of the disparity and wondered whether biases on the part of peer reviewers might explain why some application topics fare less well when submitted to the NIH for consideration of funding. In other words, peer reviewers might prefer certain topics over others, and specifically may be biased against applications focused on topics preferred by AAB PI’s.

However, peer review outcomes are not the only determinant of funding. Across the agency, grants are not simply awarded in order of peer review scores (Taffe and Gilpin, 2021). One reason is that applications submitted to the NIH are not just submitted to NIH; they are assigned to one of 24 grant-issuing institutes or centers ('ICs') that in turn make decisions about which applications to fund. The proportion of applications funded (or 'award rate') varies across ICs; therefore, we can think of the NIH process as not one competition hinging entirely on peer review but rather as 24 separate competitions. The variability of award rates relates to differences in number of applications each IC receives, available funds, and IC priorities.

Hoppe et al., 2019 did not account for IC assignment or variation in IC-specific award rates. It is possible that the apparent link between topic choice and funding disparities may reflect differences in IC assignment, since ICs receive applications according to alignment with their stated mission. For example, applications focusing on cancer epidemiology are more likely to be assigned to the National Cancer Institute while those focusing on basic human biology are more likely to be assigned to the National Institute of General Medical Sciences. If award rates at the National Institutes of General Medical Sciences are higher than at the National Cancer Institute, it might appear that NIH 'prefers' basic human biology over cancer epidemiology. While the former topic does fare better with a higher likelihood of funding, this may be largely because of different IC award rates as opposed to differences in how the topics are received by peer reviewers.

We therefore re-analyzed the data from Hoppe et al., 2019 focusing on topic-specific peer review outcomes according to AAB PI preferences and on the possible role of IC assignment in application outcomes. To minimize well-documented biases related to repeated looks by the peer review system on individual proposals (from resubmissions [Lauer, 2017] or competing renewals [Lauer, 2016]) we focus on de novo applications submitted to the NIH for the first time.

These analyses are based on R01 applications submitted to NIH between 2011 and 2015. Hoppe et al., 2019 described in detail NIH peer review processes and the 'Word2vec' algorithm Mikolov et al., 2013 used to designate a topic for each application. Briefly, each application is assigned to a peer review group; approximately 2% of applications are withdrawn prior to review for administrative reasons. After a preliminary pre-meeting review, approximately half are deemed to be potentially meritorious and are therefore discussed during a formally convened meeting. After the meeting, each discussed application receives a priority score ranging from 10 (best) to 90 (worst); many, but not all, applications also receive a 'percentile ranking' to account for differences in how individual peer review groups calibrate their scores.

Applications are not only assigned to peer review groups; they are also assigned to ICs. The Center of Scientific Review (CSR) Division of Receipt and Referral makes assignments (CSR, 2021) based on IC interests, specific Funding Opportunity Announcements (as appropriate), and sometimes on applicant request. Some applications receive a dual assignment (that is, two ICs with one primary and one secondary), but changes in primary IC assignment occur less than 2% of the time. ICs ultimately make decisions about which applications to fund, with funding decisions based on peer review scores, strategic priorities, and availability of funds (NIH, 2021c).

Regression analyses

Request a detailed protocol

Figure 1 shows the association of cumulative percentage of AAB applications by the cumulative percentage of topics. Consistent with Hoppe et al., 2019, there was a nonrandom distribution whereby 10% of topics (or 15 topics) accounted for 50% of AAB applications, while 25% of the topics accounted for 70% of AAB applications. We designted a topic as 'AAB Preferred' if it was among the 15 topics that accounted for 50% of AAB applications.

Figure 1

Download asset Open asset

To further assess the association of topic choice with peer review outcomes, we focused on applications that were discussed and therefore received a priority score. We constructed a plot of topic-specific mean peer review scores according to number of applications in each topic. We expected to find a greater variance of mean scores for topics receiving fewer applications ('regression to the mean'). We generated a linear regression model to estimate a predicted mean score for each topic based on topic size (namely the number of submitted applications that were discussed for each topic), and calculated a residual for each topic by subtracting from each topic-specific mean score the model-based predicted mean score. We compared the distribution of residuals for AAB preferred topics and other topics.

We next performed a series of probit regression analyses of application funding as

(7) ${\displaystyle Y_i={\beta }_0+{\beta }_{1}P{I}_{AAB}+{\beta }_2{T}_{AABP}+{\beta }_3{A}_{IC}+\beta \mathbf{X}+ϵ}$

where $P{I}_{AAB}$ is an indicator variable for whether application i had an African American or Black PI (yes or no), $T_{AABP}$ is an indicator variable for whether application i focused on an AAB Preferred topic (yes or no), $A_{IC}$ is the IC award rate (Equation 2), and $\mathbf{𝐗}$ corresponds to variables including 'Early Stage Investigator' status, whether application i designated more than one PI, and whether the proposed research in application i involved animal and/or human subjects. Early Stage Investigators were those PIs within 10 years of their terminal research degree or completion of clinical training. Akaike Information Criteria, Bayesian Information Criteria, and Log Likelihood values informed model strength.

All analyses used R (R Foundation, 2021) packages, including tidyverse (Wickham et al., 2019), ggplot2 (Wickham, 2016), finalfit (Harrison, 2020), and texreg (Leifeld, 2013).

Of 157,405 applications received, there were, after exclusion of pre-review withdrawals, resubmissions and competing renewals, 96,697 applications considered and reviewed by NIH for the first time. Of these 8381 were funded, for an overall award rate of 9%. The vast majority of applications (88%) were reviewed in the NIH Center for Scientific Review (or CSR). There were 1643 applications, or 2%, submitted by AAB PIs. Table 1 shows IC-specific values for applications received, applications funded, award rates (Equation 2), and percent applications coming from AAB PIs. Of note, award rates varied from 6% to 15%, while the proportion of applications with AAB PIs ranged from <1% to nearly 15%. We also show 2015 appropriations for each IC (NIH, 2021b).

Table 2 shows review and funding outcomes for applications according to whether the assignment was to an IC in the top quartile of proportion of applications with AAB PIs ('Higher AAB'). These ICs were the National Institute of Allergy and Infectious Diseases, the National Institute of Dental and Craniofacial Research, the National Institute of Child Health and Human Development, the National Institute of Minority Health and Disparities, the National Institute of Nursing Research, and the Fogarty International Center. Applications assigned to Higher AAB ICs were three times more likely to come from AAB PIs. Review outcomes – proportion discussed and, for those applications that were discussed at peer review meetings, priority scores and percentile rankings – were similar in both groups. Despite the similar review outcomes, these applications were 10% less likely to be funded. These descriptive data show that two IC-defined cohorts of applications with similar review outcomes can have substantively different funding outcomes.

Review outcomes according to topic

Figure 2 shows the topic-based peer review outcomes according to the proportion of total AAB applications linked to each topic. There was a non-significant trend whereby applications focusing on topics that accounted for a greater proportion of AAB applications (Equation 1) were more likely to make it to discussion (Equation 5)(Panel A, p=0.12). When we focused on applications that were discussed and therefore received a priority score, there was no association of mean score for each topic (Equation 6) with proportion of total AAB applications linked to each topic (Equation 1) (Panel B, p=0.87).

Figure 2

Download asset Open asset

To gain greater insights into possible peer review biases against topics preferred by AAB PIs, Figure 3, Panel A, shows the mean priority score by topic (Equation 6; again, only discussed applications receive priority scores) according to the topic size, namely the number of submitted applications that were discussed for each topic. As would be expected topics of smaller size showed greater variability, a manifestation of regression to the mean.

Figure 3

Download asset Open asset

The line in Figure 3, Panel A, is based on a linear regression of mean score according to topic size. Although the slope was slightly negative (coeffecient −0.0002634), the association was not significant (p=0.63). Among AAB preferred topics (orange dots), there were 5 more than 1 point above the line (meaning with scores worse than predicted), while there were 3 more than 1 point below the line (meaning with scores better than predicted). The remaining seven topics had mean scores that were within 1 point of the predicted value.

For each topic, we calculated a residual by subtracting from the topic-specific mean priority score (Equation 6) the predicted mean priority score; we weighted the residuals by the topic size, as the larger topics contribute more information. Figure 3, Panel B, shows the distribution of the weighted residuals according to topic type. Residuals were more positive (i.e. worse) for AAB preferred topics. However, the absolute differences were small, much less than one priority score point (over a possible range of 10–90, with topic-specific mean values ranging from 35 to 45).

Funding outcomes according to topic: descriptive findings

Table 3 shows peer review and funding outcomes according to whether applications were focused on the 15 (or 10% of) topics that made up 50% of all applications with AAB PIs ('AAB Preferred' topics). Consistent with the absence of association with proportion of AAB applications linked to each topic (Figure 2), peer review outcomes were similar in the two groups. However, applications focusing on AAB Preferred topics were 7% less likely to be funded. Table 4 shows similar findings when focusing on the 37 (or 25% of) topics that made up 70% of all applications with AAB PIs.

Table 3

Characteristic or outcome		AAB preferred	Other
Total N (%)		24416 (25.3)	72281 (74.7)
PI AAB	Yes	824 (3.4)	819 (1.1)
Discussed	Yes	11515 (47.2)	33139 (45.8)
Priority Score	Median (IQR)	40.0 (31.0 to 48.0)	39.0 (30.0 to 48.0)
Percentile Ranking	Median (IQR)	34.0 (22.0 to 45.0)	33.0 (20.0 to 44.0)
Assigned IC AAB Proportion	Higher AAB	5925 (24.3)	12337 (17.1)
Awarded	Yes	1992 (8.2)	6389 (8.8)

Table 4

Characteristic or outcome		AAB preferred	Other
Total N (%)		46108 (47.7)	50589 (52.3)
PI AAB	Yes	1152 (2.5)	491 (1.0)
Discussed	Yes	21485 (46.6)	23169 (45.8)
Priority Score	Median (IQR)	40.0 (31.0 to 48.0)	39.0 (30.0 to 48.0)
Percentile Ranking	Median (IQR)	34.0 (21.0 to 45.0)	33.0 (20.0 to 44.0)
Assigned IC AAB Proportion	Higher AAB	11424 (24.8)	6838 (13.5)
Awarded	Yes	3852 (8.4)	4529 (9.0)

Why do applications on AAB Preferred topics have worse funding outcomes despite similar peer review assessments? Table 3 shows that applications on AAB Preferred topics (those 10% of topics that accounted for 50% of all AAB applications) were 42% more likely to be assigned to Higher AAB ICs. The scatter plot in Figure 4 shows IC award rate according to the proportion of applications assigned to it that focus on AAB Preferred topics. ICs with receiving a higher percentage of AAB Preferred topic applications tended to have lower award rates (r = −0.35, p=0.08).

Figure 4

Download asset Open asset

The association between IC award rates and application success is demonstrated in Figure 5, Panel A, which shows that observed topic-specific award rates (Equation 3) were strongly correlated with predicted award rates (Equation 4), that is award rates that would be expected based solely on IC-specific award rates (r = 0.62, p<0.0001). Figure 5, Panel B shows a tendency toward lower predicted topic-specific award rates (Equation 4) with increasing proportion of AAB applications (Equation 1) linked to each topic (p=0.06). Panel C shows that the actual topic-specific award rates (Equation 3) were non-significantly negatively associated with increasing proportion of AAB applications (Equation 1) (p=0.76).

Figure 5

Download asset Open asset

Among over 96,000 R01 applications submitted to NIH between 2011 and 2015, 2% were submitted by AAB PIs. Their applications were skewed towards a relatively small group of 'AAB Preferred' topics (Figure 1); 10% of 148 topics accounted for 50% of AAB applications. Applications on AAB Preferred topics had similar review outcomes as those on other topics (Figure 2 and Table 3) but were less likely to be funded. The lower award rates for applications on AAB Preferred applications (Table 5, Model 1) were no longer seen after accounting for IC award rates (Table 6, Models 2 and 3).

These observations reflect that there are two factors at play in determining whether an application submitted to NIH will be funded. The first, well known to all involved with NIH system, is peer review; those applications that receive better scores are more likely to be funded. But there is a second factor, namely the funding ecology of the IC to which the application is assigned. As shown in Table 2 applications with similar peer review outcomes are less likely to be funded if they are assigned to ICs with lower overall award rates. AAB PIs are more likely to submit applications to ICs with lower award rates, and applications (whether submitted by AAB or other PIs) that focus on AAB Preferred topics tended to be assigned to ICs with lower award rates (Figure 4).

Hoppe et al., 2019 found that topic choice accounted for over 20% of funding disparities that adversely effect AAB PIs. We confirm this, but find that IC assignment (which, of course, is linked to topic) explains the disparities just as well, and that IC award rates explain the disparities even better (Table 5, 14% vs 5% reduction in absolute value of regression coefficient for AAB PI). Furthermore, after accounting for IC award rate, we found no association between topic choice and funding outcome (Table 6).

There is variability in how well different topics fare at peer review, but inspection of Figure 3 suggests that much of this variability reflects instability of estimates stemming from smaller sample sizes. Many topics that are not favored by AAB PIs receive better (lower) priority scores than the overall average, but many other such topics receive worse scores (Figure 3, Panel A). An inspection of weighted residuals suggest that AAB Preferred topics may fare a bit worse (Figure 3, Panel B), but to a lower degree than the difference of award rates among assigned ICs (Table 1). Furthermore, it should be noted that applications on these topics were more likely to make it past the first hurdle of peer review, that is reaching the point of formal discussion (Figure 2 and Table 3, see line 'Discussed'); thus, if anything, peer reviewers may be slightly biased in favor of AAB-preferred topics.

Our primary analysis focused on first-time submissions of de novo applications in which the award rates were low (<10%). Nonetheless, our analyses of resubmission applications yielded similar findings (Appendix 1). It is also important to note that this was an analysis of applications, not persons. This is an issue when considering multi-PI applications, since all PI’s, not just the contact PI, plays a role in choosing the topic of their proposal. Nonetheless, an analysis confined to single PI applications yielded similar findings (Appendix 2).

Only 2% of PIs in this study were African American or Black and thus it is not possible that this small percentage of scientists cover the entire breadth of what NIH funds. The small number of AAB PIs is a source of error for describing AAB scientific interests as reflected in grant applications. Nonetheless, in recent years, NIH has increasingly recognized the importance of enhancing research in targeted topics, as exemplified by a recently announced Common Fund program to enhance transformative research in health disparities and health equity (NIH, 2021a) and by the large investment in RADx-UP, a COVID-19 diagnostic testing program in underserved communities (Tromberg et al., 2020).

The authors have provided the de-identified data frame (in. RData format) along with three R markdown files that will make it possible for interested readers to reproduce the main paper and the two appendices (including all tables, figures, and numbers in the text).

1. Website

   1. CSR

   (2021) The assignment process

   Accessed April 25, 2021.

   https://public.csr.nih.gov/ForApplicants/SubmissionAndAssignment/DRR/assignmentprocess
2. 1. Ginther DK
   2. Schaffer WT
   3. Schnell J
   4. Masimore B
   5. Liu F
   6. Haak LL
   7. Kington R

   (2011) Race, ethnicity, and NIH research awards

   Science 333:1015–1019.

   https://doi.org/10.1126/science.1196783

   • PubMed
   • Google Scholar
3. Website

   1. Harrison E

   (2020) Tom Drake, and ots Quickly Create Elegant Regression Results Tables and Plots when Modelling

   Accessed April 25, 2021.

   https://finalfit.org/index.html
4. 1. Hoppe TA
   2. Litovitz A
   3. Willis KA
   4. Meseroll RA
   5. Perkins MJ
   6. Hutchins BI
   7. Davis AF
   8. Lauer MS
   9. Valantine HA
   10. Anderson JM
   11. Santangelo GM

   (2019) Topic choice contributes to the lower rate of NIH awards to African-American/black scientists

   Science Advances 5:eaaw7238.

   https://doi.org/10.1126/sciadv.aaw7238

   • PubMed
   • Google Scholar
5. Website

   1. Lauer MS

   (2016) Are attempts at renewal successful?

   Accessed April 25, 2021.

   https://nexus.od.nih.gov/all/2016/02/16/are-attempts-at-renewal-successful/
6. Website

   1. Lauer MS

   (2017) Resubmissions revisited: funded resubmission applications and their initial peer review scores

   Accessed April 25, 2021.

   https://nexus.od.nih.gov/all/2017/02/17/resubmissions-revisited-funded-resubmission-applications-and-their-initial-peer-review-scores/
7. 1. Leifeld P

   (2013) Texreg: conversion of statistical model output in R to LaTeX and HTML tables

   Journal of Statistical Software 55:1–24.

   https://doi.org/10.18637/jss.v055.i08

   • Google Scholar
8. Preprint

   1. Mikolov T
   2. Chen K
   3. Corrado G
   4. Dean J

   (2013) Efficient estimation of word representations in vector space

   arXiv.

   https://arxiv.org/abs/1301.3781

   • Google Scholar
9. Website

   1. NIH

   (2021a) ACD meetings

   Accessed April 25, 2021.

   https://www.acd.od.nih.gov/meetings.html
10. Website

    1. NIH

    (2021b) NIH — Office of Budget —Appropriations History by Institute/Center (1938 to Present)

    Accessed April 25, 2021.

    https://officeofbudget.od.nih.gov/approp_hist.html
11. Website

    1. NIH

    (2021c) NIH strategic plans and visions | RePORT

    Accessed April 25, 2021.

    https://report.nih.gov/reports/strategic-plans
12. Software

    1. R Foundation

    (2021) R: The R Project for Statistical Computing

    R: The R Project for Statistical Computing.

    https://www.r-project.org/
13. 1. Taffe MA
    2. Gilpin NW

    (2021) Racial inequity in grant funding from the US National Institutes of Health

    eLife 10:e65697.

    https://doi.org/10.7554/eLife.65697

    • Google Scholar
14. 1. Tromberg BJ
    2. Schwetz TA
    3. Pérez-Stable EJ
    4. Hodes RJ
    5. Woychik RP
    6. Bright RA
    7. Fleurence RL
    8. Collins FS

    (2020) Rapid scaling up of Covid-19 diagnostic testing in the united states — The NIH RADx Initiative

    New England Journal of Medicine 383:1071–1077.

    https://doi.org/10.1056/NEJMsr2022263

    • Google Scholar
15. Book

    1. Wickham H

    (2016) GGplot2: Elegant Graphics for Data Analysis

    New York: Springer-Verlag.

    https://doi.org/10.1007/978-0-387-98141-3

    • Google Scholar
16. 1. Wickham H
    2. Averick M
    3. Bryan J
    4. Chang W
    5. McGowan L
    6. François R
    7. Grolemund G
    8. Hayes A
    9. Henry L
    10. Hester J
    11. Kuhn M
    12. Pedersen T
    13. Miller E
    14. Bache S
    15. Müller K
    16. Ooms J
    17. Robinson D
    18. Seidel D
    19. Spinu V
    20. Takahashi K
    21. Vaughan D
    22. Wilke C
    23. Woo K
    24. Yutani H

    (2019) Welcome to the tidyverse

    Journal of Open Source Software 4:1686.

    https://doi.org/10.21105/joss.01686

    • Google Scholar

Author details

1. Michael S Lauer

   Office of the Director, National Institutes of Health, Bethesda, United States

   Contribution

   Conceptualization, Formal analysis, Supervision, Methodology, Writing - original draft, Project administration

   For correspondence

   Michael.Lauer@nih.gov

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-9217-8177

2. Jamie Doyle

   Division of Clinical Innovation, National Center for Advancing Translational Sciences, Bethesda, United States

   Contribution

   Conceptualization, Formal analysis, Methodology, Writing - review and editing

   Competing interests

   No competing interests declared

3. Joy Wang

   Office of Extramural Research, National Institutes of Health, Bethesda, United States

   Contribution

   Formal analysis, Methodology, Writing - review and editing

   Competing interests

   No competing interests declared

4. Deepshikha Roychowdhury

   Office of Extramural Research, National Institutes of Health, Bethesda, United States

   Contribution

   Formal analysis, Methodology, Writing - review and editing

   Competing interests

   No competing interests declared

• 212

  downloads

Views, downloads and citations are aggregated across all versions of this paper published by eLife.