An experimental test of the effects of redacting grant applicant identifiers on peer review outcomes

  1. Richard Nakamura
  2. Lee S Mann
  3. Mark D Lindner PhD
  4. Jeremy Braithwaite
  5. Mei-Ching Chen
  6. Adrian Vancea
  7. Noni Byrnes
  8. Valerie Durrant
  9. Bruce Reed  Is a corresponding author
  1. Retired, United States
  2. NIH/Center for Scientific Review, United States
  3. National Institutes of Health, United States
  4. Evaluact, Inc, United States

Abstract

Background: Blinding reviewers to applicant identity has been proposed to reduce bias in peer review.

Methods: This experimental test used 1200 NIH grant applications, 400 from Black investigators, 400 matched applications from White investigators, and 400 randomly selected applications from White investigators. Applications were reviewed by mail in standard and redacted formats.

Results: Redaction reduced, but did not eliminate, reviewers' ability to correctly guess features of identity. The primary, pre-registered analysis hypothesized a differential effect of redaction according to investigator race in the matched applications. A set of secondary analyses (not pre-registered) used the randomly selected applications from White scientists and tested the same interaction. Both analyses revealed similar effects: Standard format applications from White investigators scored better than those from Black investigators. Redaction cut the size of the difference by about half (e.g. from a Cohen's d of 0.20 to 0.10 in matched applications); redaction caused applications from White scientists to score worse but had no effect on scores for Black applications.

Conclusions: Grant-writing considerations and halo effects are discussed as competing explanations for this pattern. The findings support further evaluation of peer review models that diminish the influence of applicant identity.

Funding: Funding was provided by the NIH.

Data availability

All data analyzed for the findings presented in this manuscript are included in the supporting files

Article and author information

Author details

  1. Richard Nakamura

    Retired, Takoma Park, MD, United States
    Competing interests
    Richard Nakamura, now retired, was Director of the NIH Center for Scientific Review (CSR) while the study was designed and implemented..
  2. Lee S Mann

    NIH/Center for Scientific Review, Bethesda, United States
    Competing interests
    Lee S Mann, now retired, was employed by CSR..
  3. Mark D Lindner PhD

    Center for Scientific Review, National Institutes of Health, Bethesda, United States
    Competing interests
    Mark D Lindner, is employed by NIH/CSR.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-8646-2980
  4. Jeremy Braithwaite

    Evaluact, Inc, Playa Vista, CA, United States
    Competing interests
    Jeremy Braithwaite, was employed by the contract research organization that conducted the data collection and initial analysis..
  5. Mei-Ching Chen

    NIH/Center for Scientific Review, Bethesda, United States
    Competing interests
    Mei-Ching Chen, MC is employed by NIH/CSR..
  6. Adrian Vancea

    NIH/Center for Scientific Review, Bethesda, United States
    Competing interests
    Adrian Vancea, is employed by NIH/Center for Scientific Review..
  7. Noni Byrnes

    NIH/Center for Scientific Review, Bethesda, United States
    Competing interests
    Noni Byrnes, is employed by NIH/Center for Scientific Review. She is the Director of CSR..
  8. Valerie Durrant

    NIH/Center for Scientific Review, Bethesda, United States
    Competing interests
    Valerie Durrant, is employed by NIH/CSR.
  9. Bruce Reed

    NIH/Center for Scientific Review, Bethesda, United States
    For correspondence
    bruce.reed@nih.gov
    Competing interests
    Bruce Reed, is employed by NIH, he is the Deputy Director of CSR.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-1606-8646

Funding

National Institutes of Health (none)

  • Richard Nakamura

Employees of the NIH were involved in study design, in data analysis, data interpretation and manuscript writing. Data were collected, and major data analysis completed, by a contract research organization.

Reviewing Editor

  1. Mone Zaidi, Icahn School of Medicine at Mount Sinai, United States

Ethics

Human subjects: All participants gave informed consent to participate in this study in accordance with a protocol that was approved on March 27, 2017 by the Social Solutions, Inc. IRB, (FWA 00008632), protocol #47.

Version history

  1. Received: June 17, 2021
  2. Preprint posted: June 28, 2021 (view preprint)
  3. Accepted: October 8, 2021
  4. Accepted Manuscript published: October 19, 2021 (version 1)
  5. Version of Record published: November 24, 2021 (version 2)

Copyright

This is an open-access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication.

Metrics

  • 4,226
    views
  • 537
    downloads
  • 17
    citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Open citations (links to open the citations from this article in various online reference manager services)

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)

  1. Richard Nakamura
  2. Lee S Mann
  3. Mark D Lindner PhD
  4. Jeremy Braithwaite
  5. Mei-Ching Chen
  6. Adrian Vancea
  7. Noni Byrnes
  8. Valerie Durrant
  9. Bruce Reed
(2021)
An experimental test of the effects of redacting grant applicant identifiers on peer review outcomes
eLife 10:e71368.
https://doi.org/10.7554/eLife.71368

Share this article

https://doi.org/10.7554/eLife.71368

Further reading

  1. Edited by Peter A Rodgers
    Collection

    The study of science itself is a growing field of research.

    1. Cell Biology
    2. Structural Biology and Molecular Biophysics
    Marcel Proske, Robert Janowski ... Dierk Niessing
    Research Article

    Mutations in the human PURA gene cause the neurodevelopmental PURA syndrome. In contrast to several other monogenetic disorders, almost all reported mutations in this nucleic acid-binding protein result in the full disease penetrance. In this study, we observed that patient mutations across PURA impair its previously reported co-localization with processing bodies. These mutations either destroyed the folding integrity, RNA binding, or dimerization of PURA. We also solved the crystal structures of the N- and C-terminal PUR domains of human PURA and combined them with molecular dynamics simulations and nuclear magnetic resonance measurements. The observed unusually high dynamics and structural promiscuity of PURA indicated that this protein is particularly susceptible to mutations impairing its structural integrity. It offers an explanation why even conservative mutations across PURA result in the full penetrance of symptoms in patients with PURA syndrome.