1. Human Biology and Medicine
Download icon

Research: NIH peer review percentile scores are poorly predictive of grant productivity

  1. Ferric C Fang  Is a corresponding author
  2. Anthony Bowen  Is a corresponding author
  3. Arturo Casadevall  Is a corresponding author
  1. University of Washington School of Medicine, United States
  2. Albert Einstein College of Medicine, United States
  3. Johns Hopkins Bloomberg School of Public Health, United States
Feature Article
Cite this article as: eLife 2016;5:e13323 doi: 10.7554/eLife.13323
3 figures

Figures

Figure 1 with 1 supplement
Publication and citation productivity in relation to percentile score.

(A) The number of publications acknowledging support from grants within five years of grant approval (from PubMed) versus the percentile score: the bar shows the mean number of publications for all grants with that percentile score. (B) The number of citations that the papers in (A) received until the end of 2013 (data from Web of Science) versus the percentile score: the bar shows the mean number of citations for all grants with that percentile score. The lowest percentile scores are the most favorable. n = 102,740. Error bars = SDM. *Pink bars indicate significantly different from all cohorts of grants receiving poorer scores by one-way ANOVA. Black and gray bars do not differ significantly from their neighbors and are shown in different shades to allow easier visualization.

https://doi.org/10.7554/eLife.13323.002
Figure 1—figure supplement 1
Random forest model of grant percentile score as a predictor of citation productivity.

A non-parametric model was constructed with 500 trees to measure grant percentile score as a predictor of citation productivity. The results indicate that 0.98% of variance in productivity can be accounted for by percentile score. The mean of squared residuals converges to 366,620.

https://doi.org/10.7554/eLife.13323.003
Grants stratified on the basis of publication and citation productivity for different percentile scores.

Graphs showing, for percentile scores of 20 or better, the number of grants in the top half (left bar) and bottom half (right right) of grants on the basis of publications (A) and citations (B). Grants in the top half on the basis of publication productivity (A) had ≥ 6 publications: mean percentile score of top half 9.244 ± 5.583, median 9; mean percentile score of bottom half 9.947 ± 5.612, median 10. Grants in the top half on the basis of citation productivity (B) had ≥ 128 citations: mean percentile score of top half 9.242 ± 5.625, median 9; mean percentile score of bottom half 9.939 ± 5.571, median 10. Fewer grants received a percentile score of zero as a result of rounding to the nearest whole number, as well as a change in the NIH percentiling algorithm since 2009.

https://doi.org/10.7554/eLife.13323.004
Receiver operating characteristic curve of grant percentile score as a predictor of citation productivity (low/high).

Area under the curve (AUC) = 0.54 (95% confidence interval: 0.53–0.54) for citation productivity greater than the median. An AUC of 1.0 corresponds to a perfect test; an AUC of 0.5 indicates performance equivalent to random chance alone.

https://doi.org/10.7554/eLife.13323.005

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Download citations (links to download the citations from this article in formats compatible with various reference manager tools)

Open citations (links to open the citations from this article in various online reference manager services)