1. Computational and Systems Biology
Download icon

Associations of topic-specific peer review outcomes and institute and center award rates with funding disparities at the National Institutes of Health

  1. Michael S Lauer  Is a corresponding author
  2. Jamie Doyle
  3. Joy Wang
  4. Deepshikha Roychowdhury
  1. Office of the Director, National Institutes of Health, United States
  2. Division of Clinical Innovation, National Center for Advancing Translational Sciences, United States
  3. Office of Extramural Research, National Institutes of Health, United States
Research Article
Cite this article as: eLife 2021;10:e67173 doi: 10.7554/eLife.67173
9 figures, 10 tables and 5 additional files

Figures

Cumulative percentage plot showing the association of cumulative percentage of applications submitted by AAB Principal Investigators and cumulative percentage of topics.

Each dot represents a topic; the first dot on the lower left corner shows that this one topic accounted for over 8% of all applications submitted by an AAB PI. Dashed vertical lines show that 10% of the topics accounted for 50% of the applications submitted by AAB Principal Investigators and that 25% of the topics accounted for 70% of the applications submitted by AAB Principal Investigators. AAB = African American or Black.

Topic peer review outcomes according to proportion of AAB applications linked to specific topics.

AAB = African American or Black. Panel A: Scatter plot of topic-specific probability of discussion according to proportion of AAB applications. Each dot refers to a topic; the three topics furthest to the right on the X-axis correspond to the three topics at the lower left of Figure 1. The line is based on a linear regression of proportion of topic-specifc applications discussed on proportion of AAB applications. The slope of the line was not significant (p=0.12). Panel B: Scatter plot of topic-specific mean priority score according to proportion of AAB applications. Lower mean priority scores correspond to better, more favorable reviews. Each dot refers to a topic; the three topics furthest to the right on the X-axis correspond to the three topics at the lower left of Figure 1. The line is based on a linear regression of proportion of topic-specific mean priority score on proportion of AAB applications; analyses are based on those applications that were discussed. The slope of the line was not significant (p=0.87).

Topic peer review scores according to number of applications received ('Topic Size') and topic type (AAB Preferred or Other).

Panel A: Scatter plot of topic-specific mean peer review scores according to topic size. Each dot refers to a topic, with orange dots AAB preferred topics and green dots all others. The line is based on a linear regression of mean peer review scores on topic size. The slope of the line was not significant (p=0.63). Panel B: Distribution of weighted residuals of topic-specific mean review scores. Residuals are calculated as the distance between the dots in Panel A and the regression line, and are then weighted by topic size.

Scatter plot of IC-specific award rates according to proportion of IC applications that focus on AAB Preferred topics.

In other words, the X-axis value would be 33% if one-third of all IC applications focused on AAB Preferred topics, namely those 15 topics that together accounted for 50% of all applications with AAB PIs. The orange-colored ICs are those in which the proportion of applications from AAB PIs were in the top quartile. AAB = African American or Black; IC = Institute or Center; EY = National Eye Institute; DC = National Institute of Deafness and Communications Disorders; GM = National Institute of General Medical Sciences; DE = National Institute of Dental and Craniofacial Research; MH = National Institute of Mental Health; DA = National Institute on Drug Abuse; NS = National Institute of Neurological Disorders and Stroke; NR = National Institute of Nursing Research; HL = National Heart, Lung, and Blood Institute; AI = National Institute of Allergy and Infectious Diseases; ES = National Institute of Environmental Health Sciences; DK = National Institute of Diabetes and Digestive and Kidney Disease; AA = National Institute on Alcohol Abuse and Alcoholism; AG = National Institute on Aging; EB = National Institute of Biomedical Imaging and Bioengineering; CA = National Cancer Institute; HD = Eunice Kennedy Shriver National Institute of Child Health and Human Development; MD = National Institute on Minority Health and Health Disparities; AR = National Institute of Arthritis and Musculoskeletal and Skin Diseases. ICs that receive relatively more applications on AAB Preferred topics tended to have lower award rates (r = −0.35, p=0.08). Data for ICs with cell sizes not exceeding 11 are not shown due to privacy concerns.

Descriptive analyses of actual (Equation 3) and predicted (Equation 4) topic-specific award rates.

AAB = African American or Black. Panel A: Association of actual vs predicted award rates. Each dot refers to a topic. The line is based on a linear regression of actual award rate on predicted award rate (r = 0.62, p<0.0001). Panel B: Association of predicted award rate (Equation 4) with proportion of AAB applications (Equation 1). Each dot refers to a topic; the three topics furthest to the right on the X-axis correspond to the 3 topics at the lower left of Figure 1. The line is based on a linear regression of predicted award rate on percent of all AAB applications; the negative slope was borderline signifiant (p=0.06). Panel C: Association of actual award rate (Equation 3) with proportion of AAB applications (Equation 1). Each dot refers to a topic; the three topics furthest to the right on the X-axis correspond to the three topics at the lower left of Figure 1. The line is based on a linear regression of actual award rate on percent of all AAB applications; the slope was non-signficant (p=0.76).

Appendix 1—figure 1
Cumulative percentage plot showing the association of cumulative percentage of applications submitted by AAB Principal Investigators and cumulative percentage of topics.

Each dot represents a topic; the first dot on the lower left corner shows that this one topic accounted for over 6% of all applications submitted by an AAB PI. Dashed vertical lines show that 11% of the topics accounted for 50% of the applications submitted by AAB Principal Investigators. AAB = African American or Black.

Appendix 1—figure 2
Topic peer review outcomes according to proportion of AAB applications linked to specific topics.

AAB = African American or Black. Panel A: Scatter plot of topic-specific probability of discussion according to proportion of AAB applications. Each dot refers to a topic; the three topics furthest to the right on the X-axis correspond to the three topics at the lower left of Figure 1. The line is based on a linear regression of proportion of topic-specifc applications discussed on proportion of AAB applications. The slope of the line was not significant (p=0.34). Panel B: Scatter plot of topic-specific mean priority score according to proportion of AAB applications. Lower mean priority scores correspond to better, more favorable reviews. Each dot refers to a topic; the three topics furthest to the right on the X-axis correspond to the three topics at the lower left of Figure 1. The line is based on a linear regression of proportion of topic-specific mean priority score on proportion of AAB applications; analyses are based on those applications that were discussed. The slope of the line was not significant (p=0.14).

Appendix 2—figure 1
Cumulative percentage plot showing the association of cumulative percentage of applications submitted by AAB Principal Investigators and cumulative percentage of topics.

Each dot represents a topic; the first dot on the lower left corner shows that this one topic accounted for over 8% of all applications submitted by an AAB PI. Dashed vertical lines show that 11% of the topics accounted for 50% of the applications submitted by AAB Principal Investigators. AAB = African American or Black.

Appendix 2—figure 2
Topic peer review outcomes according to proportion of AAB applications linked to specific topics.

AAB = African American or Black. Panel A: Scatter plot of topic-specific probability of discussion according to proportion of AAB applications. Each dot refers to a topic; the three topics furthest to the right on the X-axis correspond to the three topics at the lower left of Figure 1. The line is based on a linear regression of proportion of topic-specifc applications discussed on proportion of AAB applications. The slope of the line was not significant (p=0.12). Panel B: Scatter plot of topic-specific mean priority score according to proportion of AAB applications. Lower mean priority scores correspond to better, more favorable reviews. Each dot refers to a topic; the three topics furthest to the right on the X-axis correspond to the three topics at the lower left of Figure 1. The line is based on a linear regression of proportion of topic-specific mean priority score on proportion of AAB applications; analyses are based on those applications that were discussed. The slope of the line was not significant (p=0.84).

Tables

Table 1
Application characteristics according to Institute or Center (IC), as ranked by award rate (Equation 2, highest to lowest).

AAB = African American or Black; PI = Principal Investigator; EY = National Eye Institute; DC = National Institute of Deafness and Communications Disorders; GM = National Institute of General Medical Sciences; DE = National Institute of Dental and Craniofacial Research; MH = National Institute of Mental Health; DA = National Institute on Drug Abuse; NS = National Institute of Neurological Disorders and Stroke; NR = National Institute of Nursing Research; HL = National Heart, Lung, and Blood Institute; AI = National Institute of Allergy and Infectious Diseases; ES = National Institute of Environmental Health Sciences; DK = National Institute of Diabetes and Digestive and Kidney Disease; AA = National Institute on Alcohol Abuse and Alcoholism; AG = National Institute on Aging; EB = National Institute of Biomedical Imaging and Bioengineering; CA = National Cancer Institute; HD = Eunice Kennedy Shriver National Institute of Child Health and Human Development; MD = National Institute on Minority Health and Health Disparities; AR = National Institute of Arthritis and Musculoskeletal and Skin Diseases. Data for ICs with cell sizes not exceeding 11 are not shown due to privacy concerns. Final columns shows 2015 appropriations in billions of dollars.

ICApplications(N)Awards(N)Award rate(%)AAB PIAAB(%)Appropriations($B)
EY202631515.5140.690.684
GM9781126612.91131.162.371
DC114514112.3131.140.405
DE128214010.9262.030.400
MH506753610.6871.721.463
DA360635910.0661.831.029
NS67866689.8771.131.605
NR11801079.1554.660.141
HL11,3259958.81711.512.998
ES21931888.6442.010.745
AI94217918.41962.084.359
DK69865387.71101.571.900
AA14231057.4161.120.447
EB17771317.4150.840.330
AG42113027.2521.231.199
CA15,90610456.62061.304.950
HD54543406.21683.081.287
MD829516.212314.840.269
AR26561616.1301.130.522
Table 2
Application review and funding outcomes according to whether Institute or Center received a higher or lower proportion of applications from AAB principal investigators.

AAB = African American or Black; PI = Principal Investigator.

Characteristic or outcomeHigher AABLower AAB
Total N (%)18262 (18.9)78435 (81.1)
PI AABYes579 (3.2)1064 (1.4)
DiscussedYes8809 (48.2)35845 (45.7)
Priority ScoreMedian (IQR)39.0 (30.0 to 48.0)40.0 (30.0 to 48.0)
Percentile RankingMedian (IQR)34.0 (21.0 to 45.0)33.0 (21.0 to 44.0)
AwardedYes1440 (7.9)6941 (8.8)
Table 3
Application review and funding outcomes according to whether topic was among those that accounted for half of all AAB applications.

Abbreviations as in Table 2.

Characteristic or outcomeAAB preferredOther
Total N (%)24416 (25.3)72281 (74.7)
PI AABYes824 (3.4)819 (1.1)
DiscussedYes11515 (47.2)33139 (45.8)
Priority ScoreMedian (IQR)40.0 (31.0 to 48.0)39.0 (30.0 to 48.0)
Percentile RankingMedian (IQR)34.0 (22.0 to 45.0)33.0 (20.0 to 44.0)
Assigned IC AAB ProportionHigher AAB5925 (24.3)12337 (17.1)
AwardedYes1992 (8.2)6389 (8.8)
Table 4
Application review and funding outcomes according to whether topic was among those that accounted for 70% of all AAB applications.

Abbreviations as in Table 2.

Characteristic or outcomeAAB preferredOther
Total N (%)46108 (47.7)50589 (52.3)
PI AABYes1152 (2.5)491 (1.0)
DiscussedYes21485 (46.6)23169 (45.8)
Priority ScoreMedian (IQR)40.0 (31.0 to 48.0)39.0 (30.0 to 48.0)
Percentile RankingMedian (IQR)34.0 (21.0 to 45.0)33.0 (20.0 to 44.0)
Assigned IC AAB ProportionHigher AAB11424 (24.8)6838 (13.5)
AwardedYes3852 (8.4)4529 (9.0)
Table 5
Probit Regression Models (regression coefficients and standard errors) with focus on the PI.

Model 1 shows the univariable association of funding success according to whether the PI is AAB. The negative non-zero cofficient indicates that applications with AAB PIs are less likely to be funded; the p-value refers to the likelihood that the difference from zero would be due to chance were the null hypothesis of no association to be true. Model 2 adjusts for topic, Model 3 adjusts for IC assignment, and Model 4 adjusts for IC award rate. Note that the absolute value for the regression coefficient linking AAB PI to funding outcome decreases with each of these adjustments, with the greatest reduction after adjusting for IC award rate. AIC = Akaike Information Criterion; BIC = Bayesian Information Criterion; Num. obs. = Number of Observations. Other abbreviations same as in Tables 1 and 2.

Model 1Model 2Model 3Model 4
Intercept−1.359***−1.349***−1.348***−1.903***
(0.006)(0.007)(0.006)(0.020)
AAB Principal Investigator−0.184***−0.174***−0.174***−0.158**
(0.049)(0.049)(0.049)(0.050)
AAB Preferred Topic−0.040**
(0.013)
IC Higher AAB Applications−0.060***
(0.015)
IC Award Rate0.061***
(0.002)
AIC56996.27956989.15656982.09256175.409
BIC57015.23857017.59457010.53056203.847
Log Likelihood−28496.140−28491.578−28488.046−28084.705
Deviance56992.27956983.15656976.09256169.409
Num. obs.96697966979669796697
  1. p***<0.001; p**<0.01; p*<0.05.

Table 6
Probit Regression Models (regression coefficients and standard errors) with focus on topic type.

Model 1 shows the univariable association of funding success according to whether the topic is AAB preferred. Model 2 shows results according to topic choice and IC award rate. Model 3 includes early stage investigator status, whether applications had multiple PIs, and whether the application included research on human subjects and/or animal subjects. AIC = Akaike Information Criterion; BIC = Bayesian Information Criterion; Num. obs. = Number of Observations. Other abbreviations same as in Tables 1 and 2.

Model 1Model 2Model 3
Intercept−1.351***−1.904***−1.869***
(0.007)(0.021)(0.027)
AAB Preferred Topic−0.044**−0.005−0.013
(0.013)(0.014)(0.015)
IC Award Rate0.061***0.057***
(0.002)(0.002)
AAB Principal Investigator−0.170***
(0.050)
Early Stage Investigator0.149***
(0.015)
Multi-PI Application0.099***
(0.015)
Human Subjects−0.056
(0.014)
Animal Subjects−0.052***
(0.014)
AIC57000.24356185.92656047.521
BIC57019.20156214.36456123.355
Log Likelihood−28498.121−28089.963−28015.760
Deviance56996.24356179.92656031.521
Num. obs.966979669796697
  1. p***<0.001; p**<0.01; p*<0.05.

Appendix 1—table 1
Application review and funding outcomes according to whether topic was among those that accounted for half of all AAB applications.

Abbreviations as in Table 2.

Characteristic or outcomeAAB preferredOther
Total N (%)8193 (25.7)23707 (74.3)
PI AABYes224 (2.7)224 (0.9)
DiscussedYes6107 (74.5)17300 (73.0)
Priority ScoreMedian (IQR)31.0 (22.0 to 40.0)31.0 (22.0 to 40.0)
Percentile RankingMedian (IQR)21.0 (9.8 to 36.0)21.0 (10.0 to 35.0)
AwardedYes2419 (29.5)7186 (30.3)
Appendix 1—table 2
Probit Regression Models (regression coefficients and standard errors) with focus on topic type.

Model 1 shows the univariable association of funding success according to whether the topic is AAB preferred. Model 2 shows results according to topic choice and IC award rate. Model 3 includes early stage investigator status, whether applications had multiple PIs, and whether the application included research on human subjects and/or animal subjects. AIC = Akaike Information Criterion; BIC = Bayesian Information Criterion; Num. obs. = Number of Observations. Other abbreviations same as in Tables 1 and 2.

Model 1Model 2Model 3
Intercept−0.515***−1.397***−1.422***
(0.009)(0.065)(0.068)
AAB Preferred Topic−0.023−0.001−0.004
(0.017)(0.017)(0.018)
IC Award Rate0.029***0.029***
(0.002)(0.002)
AAB Principal Investigator−0.171**
(0.065)
Early Stage Investigator0.221***
(0.019)
Multi-PI Application0.035
(0.020)
Human Subjects−0.020
(0.018)
Animal Subjects−0.006
(0.018)
AIC39034.47638849.76338720.539
BIC39051.21738874.87438787.502
Log Likelihood−19515.238−19421.881−19352.270
Deviance39030.47638843.76338704.539
Num. obs.319003190031900
  1. p***<0.001; p**<0.01; p*<0.05.

Appendix 2—table 1
Application review and funding outcomes according to whether topic was among those that accounted for half of all AAB applications.

Abbreviations as in Table 2.

Characteristic or outcomeAAB preferredOther
Total N (%)19134 (24.9)57842 (75.1)
PI AABYes660 (3.4)663 (1.1)
DiscussedYes8993 (47.0)26342 (45.5)
Priority ScoreMedian (IQR)40.0 (31.0 to 48.0)40.0 (30.0 to 48.0)
Percentile RankingMedian (IQR)35.0 (22.0 to 45.0)34.0 (21.0 to 44.0)
AwardedYes1541 (8.1)5015 (8.7)
Appendix 2—table 2
Probit Regression Models (regression coefficients and standard errors) with focus on topic type.

Model 1 shows the univariable association of funding success according to whether the topic is AAB preferred. Model 2 shows results according to topic choice and IC award rate. Model 3 includes early stage investigator status, whether applications had multiple PIs, and whether the application included research on human subjects and/or animal subjects. AIC = Akaike Information Criterion; BIC = Bayesian Information Criterion; Num. obs. = Number of Observations. Other abbreviations same as in Tables 1 and 2.

Model 1Model 2Model 3
Intercept−1.361***−1.903***−1.856***
(0.007)(0.023)(0.030)
AAB Preferred Topic−0.040**0.0020.008
(0.015)(0.015)(0.017)
IC Award Rate0.060***0.056***
(0.002)(0.002)
AAB Principal Investigator−0.206***
(0.057)
Early Stage Investigator0.160***
(0.016)
Human Subjects−0.070***
(0.016)
Animal Subjects−0.038*
(0.016)
AIC44830.31244152.77344026.919
BIC44848.81444180.52644091.678
Log Likelihood−22413.156−22073.386−22006.460
Deviance44826.31244146.77344012.919
Num. obs.769767697676976
p***<0.001; p**<0.01; p*<0.05

Additional files

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Download citations (links to download the citations from this article in formats compatible with various reference manager tools)

Open citations (links to open the citations from this article in various online reference manager services)