Funding: Blinding peer review
Concealing the identity of the principal investigator only partially closes the success gap between white and African American or Black researchers in NIH grant applications.
- Department of Psychiatry and the Skaggs School of Pharmacy, University of California, San Diego, United States
Every year, approximately 55,000 research grant applications are submitted to the US National Institutes of Health (NIH); of those only 11,000 will be selected for funding. NIH-funded research drives major advances in scientific knowledge, medicine and healthcare, helping to improve health, to reduce morbidity and to create economic innovation. Yet, how this funding is distributed is increasingly coming under scrutiny.
In particular, research shows that applications spearheaded by principal investigators who identify as African American or Black (AAB) do not get funded as often as those led by white researchers. From 2000 to 2006, for instance, Research Project Grant (R01) applications by AAB investigators were 42% less successful than those led by white researchers (Ginther et al., 2011). When the report highlighting this gap was first published in 2011, the Director of the NIH asserted that “the situation [was] not acceptable” (Corbyn, 2011); yet a similar 40 % reduction in success rate was reported for applications with AAB principal investigators submitted from 2011 to 2015 (Hoppe et al., 2019). AAB leaders were more likely to propose investigating questions that were less often awarded money, but they were also less likely to be funded regardless of grant topic.
Overall, this body of work suggests that AAB scientists have unequitable access to public resources, hindering the advance of knowledge – especially on health care topics of interest for communities of color (Dzirasa, 2020; Gilpin and Taffe, 2021; Harnett, 2020; Stevens et al., 2021; Taffe and Gilpin, 2021). These disparities shed light on biases that may contaminate the grant-awarding mechanism, spurring interest into whether the process could be improved. Now, in eLife, Bruce Reed and colleagues at the NIH/Center for Scientific Review – including Richard Nakamura as first author – report the impact of reviewer blinding on the funding gap (Nakamura et al., 2021).
Typically, each NIH application is first evaluated in depth by three peer scientists, who provide an initial ‘overall-impact’ score which is averaged to rank the submissions. A panel of 20–30 researchers then assembles to discuss the top half of the applications assigned to them; together, they vote on a final overall-impact score for each of these projects, following a discussion led by the three assigned reviewers. Finally, these scores are used by the 24 NIH Institutes and Centers that issue the grants to decide which projects to fund (Kienholz and Berg, 2013). Initial peer review therefore plays a major role in determining which applications will receive a grant by providing an all-critical preliminary impact score. Many have therefore proposed that blinding these reviewers to the identity of the applicants could help to potentially eliminate the disparity between white and AAB investigators.
To explore this question, Nakamura et al. obtained 400 R01 applications with AAB principal investigators submitted and reviewed in 2014–2015, as well as two comparison sets of 400 applications with white investigators. One of these two sets was randomly selected from the 26,000 applications submitted in the same period; the other was created by ensuring that the applications matched those with AAB principal investigators on several characteristics, including the preliminary impact score from the assigned reviewers. The team removed any identity information from the proposals. Both anonymized and original, unredacted applications were then peer-reviewed by different sets of researchers.
Results showed that anonymizing the applications reduced the scores for projects led by white investigators, but this manipulation did not change the scores of applications from AAB researchers. Overall, the reduction in white investigators’ scores only closed the AAB-white gap by about half. Yet, several methodological issues may limit how well these results could translate to actual NIH review processes.
First, reviewers did not meet in panels to discuss applications and vote final scores; a critical part of the NIH reviewing process was therefore not duplicated, and the impact of blinding this step cannot be determined. Second, Nakamura et al. report that 22% of reviewers ‘broke the blind’ by correctly identifying the specific principal investigator or research group leading the anonymized application. Merely removing direct identifiers from proposals may therefore not be sufficient to blind review. Finally, scores were not always replicated between the original and study reviews – they were improved for white investigators in the group matching AAB applicants’ scores. This discrepancy could imply that the reviews conducted for the study might have been done differently than during the original process.
Importantly, the work by Nakamura et al. shows that changing NIH grant evaluation to blinded review will have limited impact, one that will take place primarily through reducing the advantage of non-anonymized proposals for white investigators. White researchers from one of the groups received different scores in the study, compared to the original review, which clearly suggests that review outcomes at the NIH may not reflect objective and highly repeatable assessments of merit.
This study has critical implications for fixing the NIH funding disparity first identified in the 2011 Ginther report. So many have proposed that blinding review is a simple solution; Nakamura et al. have shown that this would be insufficient. This work, combined with identified disparities in funding of topics (Hoppe et al., 2019), should re-orient the NIH away from trying to identify singular causes, and towards applying direct fixes with immediate impact.
References
-
Toward an anti-racist approach to biomedical and neuroscience researchThe Journal of Neuroscience 41:8669–8672.https://doi.org/10.1523/JNEUROSCI.1319-21.2021
-
Race, ethnicity, and NIH research awardsScience 333:1015–1019.https://doi.org/10.1126/science.1196783
Article and author information
Author details
Publication history
- Version of Record published: November 24, 2021 (version 1)
Copyright
© 2021, Taffe
This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.
Metrics
-
- 1,495
- Page views
-
- 67
- Downloads
-
- 0
- Citations
Article citation count generated by polling the highest count across the following sources: Crossref, PubMed Central, Scopus.
Download links
A two-part list of links to download the article, or parts of the article, in various formats.
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
Funding: Blinding peer review
eLife 10:e74744.
https://doi.org/10.7554/eLife.74744
Further reading
-
- Cell Biology
- Medicine
The canonical target of the glucagon-like peptide-1 receptor (GLP-1R), Protein Kinase A (PKA), has been shown to stimulate mechanistic Target of Rapamycin Complex 1 (mTORC1) by phosphorylating the mTOR-regulating protein Raptor at Ser791 following β-adrenergic stimulation. The objective of these studies is to test whether GLP-1R agonists similarly stimulate mTORC1 via PKA phosphorylation of Raptor at Ser791 and whether this contributes to the weight loss effect of the therapeutic GLP-1R agonist liraglutide. We measured phosphorylation of the mTORC1 signaling target ribosomal protein S6 in Chinese Hamster Ovary cells expressing GLP-1R (CHO-Glp1r) treated with liraglutide in combination with PKA inhibitors. We also assessed liraglutide-mediated phosphorylation of the PKA substrate RRXS*/T* motif in CHO-Glp1r cells expressing Myc-tagged wild-type (WT) Raptor or a PKA-resistant (Ser791Ala) Raptor mutant. Finally, we measured the body weight response to liraglutide in WT mice and mice with a targeted knock-in of PKA-resistant Ser791Ala Raptor. Liraglutide increased phosphorylation of S6 and the PKA motif in WT Raptor in a PKA-dependent manner but failed to stimulate phosphorylation of the PKA motif in Ser791Ala Raptor in CHO-Glp1r cells. Lean Ser791Ala Raptor knock-in mice were resistant to liraglutide-induced weight loss but not setmelanotide-induced (melanocortin-4 receptor-dependent) weight loss. Diet-induced obese Ser791Ala Raptor knock-in mice were not resistant to liraglutide-induced weight loss; however, there was weight-dependent variation such that there was a tendency for obese Ser791Ala Raptor knock-in mice of lower relative body weight to be resistant to liraglutide-induced weight loss compared to weight-matched controls. Together, these findings suggest that PKA-mediated phosphorylation of Raptor at Ser791 contributes to liraglutide-induced weight loss.
-
- Cell Biology
- Developmental Biology
Cylicins are testis-specific proteins, which are exclusively expressed during spermiogenesis. In mice and humans, two Cylicins, the gonosomal X-linked Cylicin 1 (Cylc1/CYLC1) and the autosomal Cylicin 2 (Cylc2/CYLC2) genes, have been identified. Cylicins are cytoskeletal proteins with an overall positive charge due to lysine-rich repeats. While Cylicins have been localized in the acrosomal region of round spermatids, they resemble a major component of the calyx within the perinuclear theca at the posterior part of mature sperm nuclei. However, the role of Cylicins during spermiogenesis has not yet been investigated. Here, we applied CRISPR/Cas9-mediated gene editing in zygotes to establish Cylc1- and Cylc2-deficient mouse lines as a model to study the function of these proteins. Cylc1 deficiency resulted in male subfertility, whereas Cylc2-/-, Cylc1-/yCylc2+/-, and Cylc1-/yCylc2-/- males were infertile. Phenotypical characterization revealed that loss of Cylicins prevents proper calyx assembly during spermiogenesis. This results in decreased epididymal sperm counts, impaired shedding of excess cytoplasm, and severe structural malformations, ultimately resulting in impaired sperm motility. Furthermore, exome sequencing identified an infertile man with a hemizygous variant in CYLC1 and a heterozygous variant in CYLC2, displaying morphological abnormalities of the sperm including the absence of the acrosome. Thus, our study highlights the relevance and importance of Cylicins for spermiogenic remodeling and male fertility in human and mouse, and provides the basis for further studies on unraveling the complex molecular interactions between perinuclear theca proteins required during spermiogenesis.
