National Institutes of Health (NIH) distributes over $34 billion per year in research grants to support biomedical research at research institutions and small businesses across the United States. NIH funding is important, not only to the future of scientific discovery, but to the careers of individual scientists. Grant funding enables them to pursue their scientific studies and success in obtaining NIH funding often factors into tenure and promotion decisions. In 2011, the National Academies of Science (NAS) issued a report arguing that increased diversity in the scientific workforce is critical to ensure that the United States maintains its global leadership and competitive edge in science and technology (National Academy of Sciences, N.A.o.E, 2011). The same year, Ginther et al. reported that the likelihood of Black PIs being awarded NIH research funding between 2000 and 2006 was 55% of that of White investigators (Ginther et al., 2011). This funding gap persists (Ginther et al., 2011; Ginther et al., 2018; Hoppe et al., 2019; Erosheva et al., 2020) and the proportion of NIH research grant applicants who are Black has increased only slightly in the ensuing years, from 1.4% in Ginther’s data to 2.3% in 2020.

The largest single factor determining the probability of funding in the highly competitive NIH system is the outcome of peer review. Peer review panels meet to discuss and score applications according to scientific merit; the possibility that peer review is biased is of great concern to applicants, funding agencies, and the American public (Fox and Paine, 2019; Gropp et al., 2017; Hengel, 2017; Lerback and Hanson, 2017; Wennerås and Wold, 1997; Taffe and Gilpin, 2021). Disparities in success rates of scientific publishing (Bendels et al., 2018; Hopkins et al., 2012; Ouyang et al., 2018) and in obtaining grant awards (Ginther et al., 2011; Pohlhaus et al., 2011; Ginther et al., 2016) raise questions of whether reviewer bias on applicant demographics (race, gender, age, career stage, etc.) or institutional reputation unfairly influence the review outcomes (Wahls, 2019; Witteman et al., 2019; Murray et al., 2016). Concerns are particularly salient for NIH because the criteria for evaluating the scientific merit of research projects include ‘investigators’ and ‘environment’, thus explicitly directing reviewers to take these factors into account.

It should be understood that NIH funding is not determined by peer review alone, but rather is additionally determined by scientific priorities and budgets at the funding institutes. Funding rates for major research grants vary approximately threefold, from about (10% to nearly 30%) across the institutes meaning that applications in some areas of science enjoy greater success than others. A recent paper focused attention on the finding that funding success varied substantially depending on scientific topic, and that the topics most often studied by Black investigators tend to have low funding rates (Hoppe et al., 2019). An important follow-up paper showed that this association was primarily attributable to the disparate funding rates across the 24 NIH institutes, rather than topical bias in peer review (Lauer et al., 2021). Nonetheless, peer review outcomes are a fundamental determinant of success across the NIH.

Various approaches to reducing demographic bias in review have been proposed. Blinding reviewers (in ‘double blind’ or ‘dual anonymous’ reviews) to applicant identity and institutional affiliations is one such approach (Cox and Montgomerie, 2019; Fisher et al., 1994; Haffar et al., 2019; Okike et al., 2016; Snodgrass, 2006). The literature examining the impact of blinding on review outcomes is mixed. With respect to gender, for example, blinding has been reported to reduce disparities (Budden et al., 2008; Terrell et al., 2017; Aloisi and Reid, 2019) but has also been ineffectual (Primack, 2009; Whittaker, 2008; Blank, 1991; Ledin et al., 2007). To our knowledge, there are no studies evaluating real review of scientific grants blinded with respect to race. Even so, the importance of diversifying science, the strong correlation between review and funding outcomes, and the perceived tractability of review make review interventions attractive. Strong concerns about the potential of demographic bias, and especially anti-Black racial bias, in peer review remain (Stevens et al., 2021).

The present study was part of the NIH response to Ginther’s 2011 report of the Black-White funding gap and the NAS report (National Academy of Sciences, N.A.o.E, 2011) on the lack of diversity in the U.S. scientific workforce. An NIH Advisory Committee to the Director (ACD) Working Group on Diversity in the Biomedical Research Workforce (WGDBRW) recommended that ‘NIH should design an experiment to determine the effects of anonymizing applications…’ in peer review (Working Group on Diversity in the Biomedical Research Workforce (WGDBRW) and T.A.C.t.t.D. (ACD), 2012). An NIH Subcommittee on Peer Review (National Institutes of Health, 2013) and an internal NIH group led by then-CSR Director (RN) designed it as a large-sample experimental investigation of the potential effects of racial bias on review outcomes, specifically bias against Black or African American investigators (see Figure 1).

Figure 1

Download asset Open asset

The decision to restrict the focus to Black scientists stemmed from three considerations. First, although funding gaps have been reported for other disadvantaged groups, none has been persistently as large as that experienced by Blacks. For example, in 2013 NIH award rates for major research projects for Hispanic scientists were 81% that of White investigators, and award rates for Asians 83% that of Whites. Both rates improved to 88% that of Whites in 2020. Past differences in funding rates for male versus female applicants for major NIH research awards have disappeared (https://report.nih.gov/nihdatabook/report/131). Second, the funding disparity for Blacks occurred in the context of the U.S. history of centuries long, pernicious and pleiotropic anti-Black racism. Lastly, the scale of a properly powered experiment to investigate multiple forms of demographic bias was prohibitive from a practical perspective.

The experimental intervention was to anonymize applications by post-submission redaction of applicant identity and institutional affiliation. Using real, previously reviewed NIH R01 applications, the experiment compared scores for applications from Black vs. White applicants as re-reviewed for this study in their standard (original) vs. redacted formats. The primary research question was ‘Does concealing the race and identity of the applicant affect reviewers’ overall impact scores of applications from Black and White applicants differently?’.

Designed as a test of whether blinded review reduces Black-White disparities in peer review (Ginther et al., 2011; Hoppe et al., 2019; Erosheva et al., 2020), the data are also pertinent to understanding the basis of the advantage that applications from White PIs enjoy in review. The experimental intervention, post-submission administrative redaction of identifying information, reduced but did not entirely eliminate reviewer knowledge of applicant identity. Applications submitted by Black PIs were compared to two samples of applications from White PIs, one matched on review-relevant variables, the other selected randomly. The preregistered analysis defined the primary question to be whether redaction differentially affected scores according to race and specified that it be tested using the matched White set of applications. That interaction term was statistically nonsignificant. A secondary test, using the randomly selected set of White applications and a different modeling approach, was statistically significant. We suggest that it is more useful to focus on the concordant patterns of observed data and overall similarity of the modeled results in the two analyses, rather than on small differences in significance levels.

The following effects were consistent across both samples and both modeling approaches: (1) applications from White PIs scored better than those from Black PIs; (2) standard format applications scored better than redacted; (3) redaction produced worse scores for applications from White PIs but did not change scores for applications from Black PIs. In both the primary and secondary comparisons, redaction reduced the difference in mean scores of Black and White application sets by about half (Table 6, Table 8). Thus, the data suggest that redaction, on average, does not improve review outcomes for Black PIs but does reduce the advantage that applications from White PIs have in review of standard format applications. Why?

Applications from White PIs tended to score better than applications from Black PIs. This was unexpected when comparing the matched application sets because the samples were closely matched on the scores they had received in actual NIH review. However, it was not surprising that applications from Black PIs scored worse, on average, than randomly selected applications from White PIs. A persistent gap on the order of 50% between award rates for NIH R01 grants to Black versus White PIs has been previously reported (e.g., Ginther et al., 2011; Hoppe et al., 2019; Erosheva et al., 2020), and the correlation between overall impact scores and probability of funding, across NIH, is the same for Black and White investigators (Hoppe et al., 2019). The secondary models identified several factors that partially account for the racial difference in scores. Competitive renewals of previously funded projects and resubmissions of previously reviewed applications scored better. These effects are well known and tend to favor applications from White PIs (because the representation of White PIs among established investigators is higher). Conversely, ESI status and being associated with an institution that is at the lowest end of the NIH funding distribution were both associated with worse review outcomes. Together these factors tend to disadvantage Black PIs but do not entirely account for the gap in scores.

Other studies have identified additional contributors to differential racial outcomes, including cumulative disadvantage (Ginther et al., 2011), and differences in publication citation metrics (Ginther et al., 2018). Citation metrics are associated with many factors (Tahamtan et al., 2016), some of which are not linked to the quality of the paper, factors such as differences in publication practices between areas of research (Piro et al., 2013), scientific networks (Li et al., 2019), coauthors’ reputations (Petersen et al., 2014), the Matthew effect (Wang, 2014), and race of the authors (Ginther et al., 2018).

The data reveal little evidence of systematic bias based on knowledge of, or impressions of PI race per se. NIH applications do not have fields for PI race (or gender). In this study reviewers were asked to guess PI race, after reading and scoring the application, not before, and reviewers were generally not very confident of their guesses (see Table 3). Thus, how much ‘knowledge’ reviewers had about PI race, and at what point in the process they formed their impression of race is unclear and likely varies across applications and reviewers. With standard applications, reviewers were more likely to guess that Black PIs were Black than White (58% vs. 36%), but with redacted application more likely to guess they were White than Black (61% vs. 28%); even so, redaction did not change scores for Black PIs. Conversely, redaction did not change the frequency of reviewer guesses that White PIs were White (93% standard, 87% redacted), but redaction did change scores for applications from White PIs. A reviewer’s guess that the PI was Black had a very small effect on scores (improving them), controlling for multiple other factors including actual PI race. Interpretation of this effect is statistically and substantively complex. It does not necessarily represent positive racial bias toward Black applicants. Reviewers had reason to presume that they were participating in a study examining the effects of PI race on review outcomes; in this context some might have tried to avoid appearing prejudiced and thus scored PIs they believed to be Black more favorably. It could also be that reviewers judged the science to be better because the PI was perceived to be Black in certain scientific contexts, for example, for an implementation study that hinged on engaging minority communities.

Redacted applications scored worse on average. This is not surprising given that redaction was done administratively, post-submission. The application the reviewers read was not the one written and information lost in redaction may have been important. Retrospectively redacting applications does not simply remove information but also changes the context of the information that remains. Applicants wrote their applications believing that reviewers would be given their names and institutional affiliations and that the other information in the application would be judged in that context. They also, presumably, took into account the fact that NIH grant review criteria include ‘investigators’ and ‘environment’, and that these criteria are supposed to be factored into reviewers’ final scores. Approximately 28% of reviewers of anonymized applications disagreed with the statement ‘I believe that reviewers can provide a fair, thorough, and competent review if applications are anonymized’. Alternatively, or in addition, redacted applications might have done worse because they reduced halo effects (Kaatz et al., 2014; Crane, 1967) that had inflated the scores of standard applications. Halo effects refer to the tendency to rate something better based on a general good impression of the applicant as opposed to specific relevant information; for example, scoring on the basis of positive reputation rather than the application itself. Absent halo effects, the applications may have scored worse when judged on their merits.

Thus, we believe there are two plausible explanations for why applications from White PIs did worse when redacted. One is that redaction reduced positive halo effects of PI and institution. The interconnected factors of PI reputation, scientific networks, scientific pedigree, and institutional prestige certainly can influence review. They are deeply intertwined with race and tend to favor White PIs over others. If redaction reduced halo effects, it would suggest that blinded review models might improve fairness (Ross et al., 2006; Nielsen et al., 2021). On the other hand, it may have been that when PI identity was deleted, the scientific narrative lost context and was consequently degraded. This would presumably predominantly affect more senior, established PIs, who are disproportionately White. If the effect of redaction represents a mismatch between writing and review conditions, blinding would not likely have a lasting effect because scientists will adjust their grant writing to conform to new review conditions.

There are practical problems with administrative redaction as an anonymization strategy. Each application took 2–8 hr to redact and quality assurance an additional 2–4 hr. Despite careful removal of names and other information, redaction was only partially successful in blinding reviewers to PI characteristics. Reviewers of redacted applications correctly identified PI race 70% of the time overall and were able to name the PI or research group in 22% of cases. This result, which is consistent with prior attempts at redaction, suggests that it is not possible to administratively obscure identity in all cases.

What accounts for the unexpected finding that the ‘matched’ samples of applications from Black and White PIs (selected to match on overall impact scores from their actual NIH reviews) scored differently when those same applications were reviewed as part of this study? We think it likely that the change in White matched scores represents regression to the mean, a problem that is more likely when, as is true here, the groups differ on multiple factors that differ in the same direction (Campbell and Stanley, 1963). The set of applications from Black PIs represented almost the entire population of such applications. The matched applications from White PIs were drawn from a population of 26,000 applications which, on average, scored better than the Black applications. Each observed score has a true score component and an error component which is presumed to be random. Selecting applications to match a score worse than the population mean risks selecting applications where the error terms are not random but rather are skewed negatively (making the observed scores worse). When the selected applications were re-reviewed, the error terms would be expected to be random rather than predominantly negative, and thus the mean of the observed scores would improve.

Another possibility is that differences in the conduct of real and study reviews account for the difference. The SROs for the experiment included several naïve to NIH review and the procedures for characterizing reviewer expertise and matching reviewers to applications were much simplified compared to the general practices of CSR SROs. Study reviewers saw many fewer applications than is typical for study section reviewers (~3 for most study reviewers, ~8 for CSR study sections). Because of this, any bias that affects the ranking of applications reviewers apply to ‘their pile’ would not likely be seen in this study. Also, reviewers knew that they were participating in an anonymization study and many likely suspected their scores would be used to assess bias in peer review. A complete listing of the deviations in the study procedures from the actual NIH peer review process are detailed in Supplementary file 1D. Despite these differences in review practices, the overall distribution of scores obtained in the experiment closely approximates the distribution of preliminary overall impact scores seen in the actual NIH reviews of these applications.

Designed as a test of whether blinding reviewers to applicant demographics reduces racial disparities in review outcomes, strengths of the study include a large sample (1200 applications, 2116 reviewers, 7155 reviews), the use of real NIH grant applications, and experienced NIH reviewers. Redaction of institutional and individual identity elements from applications changed scores for White PIs applications for the worse, but did not, on average, affect scores of Black PIs’ applications. Although the effect was statistically small, both samples of White PI’s applications scored better than the Black PIs applications; in each case redaction reduced the size of that difference by about half. It is possible that redaction highlighted gaps in applications written with the assumption that reviewers would know the PI identity; for example, a field-leader in use of a technique might have intended their name to substitute for methodological details. If that sort of grantsmanship accounts for the redaction effect, implementing partial blinding in review is unlikely to have any lasting benefit. If, however, blinding reduces halo effects and that accounts for the reduction in White advantage, then changing review models could perhaps result in a fairer peer review process.

Post-submission administrative redaction is too labor-intensive to implement on the scale that NIH requires. And, even this careful redaction was quite imperfect in concealing elements of identity. However, there are other methods of blinding reviewers to applicant demographics and institutional affiliations. For example, self-redaction of applications might be more effective, and two stage models of review that require judging the merit of an application’s science while blinded to applicant identity are interesting. Development of strategies to ensure a level playing field in the scientific peer review process of scientific grant applications is an urgent need.

All data analyzed for the findings presented in this manuscript are included in the supporting files.

1. Preprint

   1. Aloisi A
   2. Reid N

   (2019) Un)Conscious Bias in the Astronomical Profession: Universal Recommendations to Improve Fairness, Inclusiveness, and Representation

   arXiv.

   https://arxiv.org/abs/1907.05261

   • Google Scholar
2. 1. Bendels MHK
   2. Müller R
   3. Brueggmann D
   4. Groneberg DA

   (2018) Gender disparities in high-quality research revealed by Nature Index journals

   PLOS ONE 13:e0189136.

   https://doi.org/10.1371/journal.pone.0189136

   • PubMed
   • Google Scholar
3. 1. Blank RM

   (1991)

   The Effects of Double-Blind versus Single-Blind Reviewing: Experimental Evidence from The American Economic Review

   The American Economic Review 81:1041–1067.

   • Google Scholar
4. 1. Budden A
   2. Tregenza T
   3. Aarssen L
   4. Koricheva J
   5. Leimu R
   6. Lortie C

   (2008) Double-blind review favours increased representation of female authors

   Trends in Ecology & Evolution 23:4–6.

   https://doi.org/10.1016/j.tree.2007.07.008

   • Google Scholar
5. Book

   1. Campbell DT
   2. Stanley JC

   (1963)

   Experimental and Quasi-Experimental Designs for Research

   Houghton Mifflin Company.

   • Google Scholar
6. 1. Cox AR
   2. Montgomerie R

   (2019) The cases for and against double-blind reviews

   PeerJ 7:e6702.

   https://doi.org/10.7717/peerj.6702

   • PubMed
   • Google Scholar
7. 1. Crane D

   (1967)

   Gatekeepers of Science - Some Factors Affecting Selection of Articles for Scientific Journals

   American Sociologist 2:195–201.

   • Google Scholar
8. 1. Erosheva EA
   2. Grant S
   3. Chen MC
   4. Lindner MD
   5. Nakamura RK
   6. Lee CJ

   (2020) NIH peer review: Criterion scores completely account for racial disparities in overall impact scores

   Science Advances 6:eaaz4868.

   https://doi.org/10.1126/sciadv.aaz4868

   • PubMed
   • Google Scholar
9. 1. Fisher M
   2. Friedman SB
   3. Strauss B

   (1994)

   The effects of blinding on acceptance of research papers by peer review

   JAMA 272:143–146.

   • PubMed
   • Google Scholar
10. 1. Fox CW
    2. Paine CET

    (2019) Gender differences in peer review outcomes and manuscript impact at six journals of ecology and evolution

    Ecology and Evolution 9:3599–3619.

    https://doi.org/10.1002/ece3.4993

    • Google Scholar
11. 1. Ginther DK
    2. Schaffer WT
    3. Schnell J
    4. Masimore B
    5. Liu F
    6. Haak LL
    7. Kington R

    (2011) Race, ethnicity, and NIH research awards

    Science 333:1015–1019.

    https://doi.org/10.1126/science.1196783

    • PubMed
    • Google Scholar
12. 1. Ginther DK
    2. Kahn S
    3. Schaffer WT

    (2016) Gender, Race/Ethnicity, and National Institutes of Health R01 Research Awards: Is There Evidence of a Double Bind for Women of Color?

    Academic Medicine 91:1098–1107.

    https://doi.org/10.1097/ACM.0000000000001278

    • PubMed
    • Google Scholar
13. 1. Ginther DK
    2. Basner J
    3. Jensen U
    4. Schnell J
    5. Kington R
    6. Schaffer WT

    (2018) Publications as predictors of racial and ethnic differences in NIH research awards

    PLOS ONE 13:e0205929.

    https://doi.org/10.1371/journal.pone.0205929

    • PubMed
    • Google Scholar
14. 1. Gropp RE
    2. Glisson S
    3. Gallo S
    4. Thompson L

    (2017) Peer Review: A System under Stress

    Bioscience 67:407–410.

    https://doi.org/10.1093/biosci/bix034

    • Google Scholar
15. 1. Haffar S
    2. Bazerbachi F
    3. Murad MH

    (2019) Peer Review Bias: A Critical Review

    Mayo Clinic Proceedings 94:670–676.

    https://doi.org/10.1016/j.mayocp.2018.09.004

    • PubMed
    • Google Scholar
16. Book

    1. Hengel E

    (2017)

    Publishing While Female: Are Women Held to Higher Standards? Evidence from Peer Review

    Cambridge Working Papers in Economics.

    • Google Scholar
17. 1. Hopkins AL
    2. Jawitz JW
    3. McCarty C
    4. Goldman A
    5. Basu NB

    (2012) Disparities in Publication Patterns by Gender, Race and Ethnicity Based on a Survey of a Random Sample of Authors

    Scientometrics 1:515–534.

    https://doi.org/10.1007/s11192-012-0893-4

    • Google Scholar
18. 1. Hoppe TA
    2. Litovitz A
    3. Willis KA
    4. Meseroll RA
    5. Perkins MJ
    6. Hutchins BI
    7. Davis AF
    8. Lauer MS
    9. Valantine HA
    10. Anderson JM
    11. Santangelo GM

    (2019) Topic choice contributes to the lower rate of NIH awards to African-American/black scientists

    Science Advances 5:eaaw7238.

    https://doi.org/10.1126/sciadv.aaw7238

    • PubMed
    • Google Scholar
19. 1. Kaatz A
    2. Gutierrez B
    3. Carnes M

    (2014) Threats to objectivity in peer review: the case of gender

    Trends in Pharmacological Sciences 35:371–373.

    https://doi.org/10.1016/j.tips.2014.06.005

    • PubMed
    • Google Scholar
20. 1. Lauer MS
    2. Doyle J
    3. Wang J
    4. Roychowdhury D

    (2021) Associations of topic-specific peer review outcomes and institute and center award rates with funding disparities at the National Institutes of Health

    eLife 10:e67173.

    https://doi.org/10.7554/eLife.67173

    • PubMed
    • Google Scholar
21. 1. Ledin A
    2. Bornmann L
    3. Gannon F
    4. Wallon G

    (2007) A persistent problem Traditional gender roles hold back female scientists

    EMBO Reports 8:982–987.

    https://doi.org/10.1038/sj.embor.7401109

    • PubMed
    • Google Scholar
22. 1. Leon AC
    2. Heo M

    (2009) Sample Sizes Required to Detect Interactions between Two Binary Fixed-Effects in a Mixed-Effects Linear Regression Model

    Computational Statistics & Data Analysis 53:10.

    https://doi.org/10.1016/j.csda.2008.06.010

    • Google Scholar
23. 1. Lerback J
    2. Hanson B

    (2017) Journals invite too few women to referee

    Nature 541:455–457.

    https://doi.org/10.1038/541455a

    • Google Scholar
24. 1. Li W
    2. Aste T
    3. Caccioli F
    4. Livan G

    (2019) Early coauthorship with top scientists predicts success in academic careers

    Nature Communications 10:S41467-019-13130-4.

    https://doi.org/10.1038/s41467-019-13130-4

    • Google Scholar
25. 1. Murray DL
    2. Morris D
    3. Lavoie C
    4. Leavitt PR
    5. MacIsaac H
    6. Masson MEJ
    7. Villard M-A

    (2016) Bias in Research Grant Evaluation Has Dire Consequences for Small Universities

    PLOS ONE 11:e0155876.

    https://doi.org/10.1371/journal.pone.0155876

    • PubMed
    • Google Scholar
26. Book

    1. National Academy of Sciences, N.A.o.E

    (2011)

    Expanding Underrepresented Minority Participation: America’s Science and Technology Talent at the Crossroads

    National Academy Press.

    • Google Scholar
27. Website

    1. National Institutes of Health

    (2013) ACD Diversity Working Group Subcommittee on Peer Review

    Accessed October 21, 2021.

    https://www.acd.od.nih.gov/working-groups/prsub.html
28. 1. Nielsen MW
    2. Baker CF
    3. Brady E
    4. Petersen MB
    5. Andersen JP

    (2021) Weak evidence of country- and institution-related status bias in the peer review of abstracts

    eLife 10:e64561.

    https://doi.org/10.7554/eLife.64561

    • PubMed
    • Google Scholar
29. 1. Okike K
    2. Hug KT
    3. Kocher MS
    4. Leopold SS

    (2016) Single-blind vs Double-blind Peer Review in the Setting of Author Prestige

    JAMA 316:1315.

    https://doi.org/10.1001/jama.2016.11014

    • Google Scholar
30. 1. Ouyang D
    2. Sing D
    3. Shah S
    4. Hu J
    5. Duvernoy C
    6. Harrington RA
    7. Rodriguez F

    (2018) Sex Disparities in Authorship Order of Cardiology Scientific Publications

    Circulation. Cardiovascular Quality and Outcomes 11:e005040.

    https://doi.org/10.1161/CIRCOUTCOMES.118.005040

    • PubMed
    • Google Scholar
31. 1. Petersen AM
    2. Fortunato S
    3. Pan RK
    4. Kaski K
    5. Penner O
    6. Rungi A
    7. Riccaboni M
    8. Stanley HE
    9. Pammolli F

    (2014) Reputation and impact in academic careers

    PNAS 111:15316–15321.

    https://doi.org/10.1073/pnas.1323111111

    • PubMed
    • Google Scholar
32. 1. Piro FN
    2. Aksnes DW
    3. Rørstad K

    (2013) A macro analysis of productivity differences across fields: Challenges in the measurement of scientific publishing

    Journal of the American Society for Information Science and Technology 64:307–320.

    https://doi.org/10.1002/asi.22746

    • Google Scholar
33. 1. Pohlhaus JR
    2. Jiang H
    3. Wagner RM
    4. Schaffer WT
    5. Pinn VW

    (2011) Sex differences in application, success, and funding rates for NIH extramural programs

    Academic Medicine 86:759–767.

    https://doi.org/10.1097/ACM.0b013e31821836ff

    • PubMed
    • Google Scholar
34. 1. Primack RB

    (2009) Do gender, nationality, or academic age affect review decisions? An analysis of submissions to the journal Biological Conservation

    Biological Conservation 142:2415–2418.

    https://doi.org/10.1016/j.biocon.2009.06.021

    • Google Scholar
35. 1. Ross JS
    2. Gross CP
    3. Desai MM
    4. Hong Y
    5. Grant AO
    6. Daniels SR
    7. Hachinski VC
    8. Gibbons RJ
    9. Gardner TJ
    10. Krumholz HM

    (2006) Effect of blinded peer review on abstract acceptance

    JAMA 295:1675–1680.

    https://doi.org/10.1001/jama.295.14.1675

    • PubMed
    • Google Scholar
36. Book

    1. Rubin DB

    (2006) Matched Sampling for Causal Effects

    Cambridge University Press.

    https://doi.org/10.1017/CBO9780511810725

    • Google Scholar
37. 1. Snodgrass R

    (2006) Single- versus double-blind reviewing: an analysis of the literature

    SIGMOD Record 35:8–21.

    https://doi.org/10.1145/1168092.1168094

    • Google Scholar
38. 1. Stevens KR
    2. Masters KS
    3. Imoukhuede PI
    4. Haynes KA
    5. Setton LA
    6. Cosgriff-Hernandez E
    7. Lediju Bell MA
    8. Rangamani P
    9. Sakiyama-Elbert SE
    10. Finley SD
    11. Willits RK
    12. Koppes AN
    13. Chesler NC
    14. Christman KL
    15. Allen JB
    16. Wong JY
    17. El-Samad H
    18. Desai TA
    19. Eniola-Adefeso O

    (2021) Fund Black scientists

    Cell 184:561–565.

    https://doi.org/10.1016/j.cell.2021.01.011

    • Google Scholar
39. 1. Taffe MA
    2. Gilpin NW

    (2021) Racial inequity in grant funding from the US National Institutes of Health

    eLife 10:e65697.

    https://doi.org/10.7554/eLife.65697

    • PubMed
    • Google Scholar
40. 1. Tahamtan I
    2. Safipour Afshar A
    3. Ahamdzadeh K

    (2016) Factors affecting number of citations: a comprehensive review of the literature

    Scientometrics 107:1195–1225.

    https://doi.org/10.1007/s11192-016-1889-2

    • Google Scholar
41. 1. Terrell J
    2. Kofink A
    3. Middleton J
    4. Rainear C
    5. Murphy-Hill E
    6. Parnin C
    7. Stallings J

    (2017) Gender differences and bias in open source: pull request acceptance of women versus men

    PeerJ Computer Science 3:e111.

    https://doi.org/10.7717/peerj-cs.111

    • Google Scholar
42. 1. Wahls WP

    (2019) Opinion: The National Institutes of Health needs to better balance funding distributions among US institutions

    PNAS 116:13150–13154.

    https://doi.org/10.1073/pnas.1909217116

    • Google Scholar
43. 1. Wang J

    (2014) Unpacking the Matthew effect in citations

    Journal of Informetrics 8:329–339.

    https://doi.org/10.1016/j.joi.2014.01.006

    • Google Scholar
44. 1. Wennerås C
    2. Wold A

    (1997) Nepotism and sexism in peer-review

    Nature 387:341–343.

    https://doi.org/10.1038/387341a0

    • Google Scholar
45. 1. Whittaker RJ

    (2008) Journal review and gender equality: a critical comment on Budden et al

    Trends in Ecology & Evolution 23:478–479.

    https://doi.org/10.1016/j.tree.2008.06.003

    • Google Scholar
46. 1. Witteman HO
    2. Hendricks M
    3. Straus S
    4. Tannenbaum C

    (2019) Are gender gaps due to evaluations of the applicant or the science? A natural experiment at a national funding agency

    Lancet 393:531–540.

    https://doi.org/10.1016/S0140-6736(18)32611-4

    • Google Scholar
47. Report

    1. Working Group on Diversity in the Biomedical Research Workforce (WGDBRW) and T.A.C.t.t.D. (ACD)

    (2012)

    Draft Report of the Advisory Committee to the Director Working Group on Diversity in the Biomedical Research Workforce

    National Institutes of Health.

    • Google Scholar

Author details

1. Richard K Nakamura

   Retired, formerly Center for Scientific Review, National Institutes of Health, Bethesda, United States

   Contribution

   Conceptualization, Data curation, Funding acquisition, Project administration, Resources, Writing – review and editing

   Competing interests

   now retired, was Director of the NIH Center for Scientific Review (CSR) while the study was designed and implemented.

2. Lee S Mann

   Retired, formerly Center for Scientific Review, National Institutes of Health, Bethesda, United States

   Contribution

   Conceptualization, Writing – review and editing

   Competing interests

   now retired, was employed by CSR.

3. Mark D Lindner

   Center for Scientific Review, National Institutes of Health, Bethesda, United States

   Contribution

   Conceptualization, Formal analysis, Writing – review and editing

   Competing interests

   is employed by NIH/CSR

   "This ORCID iD identifies the author of this article:" 0000-0002-8646-2980

4. Jeremy Braithwaite

   Social Solutions International, Rockville, United States

   Present address

   EvaluACT, Inc, Playa Vista, United States

   Contribution

   Conceptualization, Data curation, Formal analysis, Writing – review and editing

   Competing interests

   was employed by the contract research organization that conducted the data collection and initial analysis.

5. Mei-Ching Chen

   Center for Scientific Review, National Institutes of Health, Bethesda, United States

   Contribution

   Data curation, Formal analysis, Conceptualization

   Competing interests

   MC is employed by NIH/CSR.

6. Adrian Vancea

   Center for Scientific Review, National Institutes of Health, Bethesda, United States

   Contribution

   Formal analysis, Conceptualization

   Competing interests

   is employed by NIH/Center for Scientific Review.

7. Noni Byrnes

   Center for Scientific Review, National Institutes of Health, Bethesda, United States

   Contribution

   Conceptualization, Resources, Writing – review and editing

   Competing interests

   is employed by NIH/Center for Scientific Review. She is the Director of CSR.

8. Valerie Durrant

   Center for Scientific Review, National Institutes of Health, Bethesda, United States

   Contribution

   Conceptualization, Writing – review and editing

   Competing interests

   is employed by NIH/CSR

9. Bruce Reed

   Center for Scientific Review, National Institutes of Health, Bethesda, United States

   Contribution

   Conceptualization, Supervision, Writing – original draft

   For correspondence

   bruce.reed@nih.gov

   Competing interests

   is employed by NIH, he is the Deputy Director of CSR

   "This ORCID iD identifies the author of this article:" 0000-0002-1606-8646

• 4,240

  views

• 537

  downloads

• 17

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.