Comparing the outputs of intramural and extramural grants funded by National Institutes of Health
- Information School, University of Wisconsin-Madison, United States
Figures
Figure 1
Research topics for intramural and extramural projects.
The topics listed were identified by clustering projects based on their titles and abstracts via Word2Vec (see Methods). The relative ratio of intramural projects for each topic was calculated by taking a ratio of the proportions of total grants a topic represented in the intramural vs. extramural portfolios. A relative ratio >1 signifies a higher share of intramural projects on that topic relative to their share across all topics. For example, if a topic comprised 10% of grants in the intramural portfolio but only 5% of grants in the extramural portfolio, this would represent a 2:1 intramural:extramural relative ratio, or 2.0. Data and code are available at Dryad (Zheng et al., 2026).
Figure 2
Project funding for intramural and extramural projects.
Mean project cost (on a log scale) vs. year for intramural grants (green) and extramural grants (blue) between 2009 and 2019. Error bars denote 95% confidence intervals. Total costs were used rather than only direct costs in order to fully account for the degree of government investment. Error bars are larger for intramural data because of the smaller total number of awards (98,648 extramural and 1594 intramural). Data and code are available at Dryad (Zheng et al., 2026).
Figure 3
Annual outputs from intramural and extramural projects.
(A) Mean number of papers per project for intramural projects (green) and extramural projects (blue) between 2008 and 2020. The difference (inset) was close to 1.5 papers per project in 2008, but this gap closed over time. (B) Relative Citation Ratio per project. (C) Approximate Potential to Translate (APT) per project. (D) Clinical citation counts per project. (E) Number of papers with at least one clinical citation per project. Error bars denote 95% confidence intervals. Error bars are larger for intramural data because of the smaller total number of awards (98,648 extramural and 1594 intramural). Data and code are available at Dryad (Zheng et al., 2026).
Figure 4
Cost effectiveness of intramural and extramural projects.
(A) A measure of cost effectiveness vs. progression (i.e., year of grant) for intramural research (green) and extramural research (blue), for projects (98,648 extramural and 1594 intramural) of different durations: 1–3 years (top row), 4–6 years (middle), and 7–10 years (bottom). These regressions do not control for other characteristics, but rather represent the raw ratios. For the first column, the Y-axis displays log10(ratio) + 1, where ratio is the cumulative total costs to the cumulative total research output for each metric (cost:output, for the first column output = #papers); error bars denote the 95% confidence intervals. The remaining columns show measures of cost effectiveness for Relative Citation Ratio, Approximate Potential to Translate, total clinical citation counts, and a binary measure of clinical citations. To account for the fact that many papers are published after funding for the relevant grant has ended, grant amounts were multiplied by a deflator – this represents the proportion of papers published to date against the anticipated number of future publications, as determined by empirical measurements (Appendix 1—table 1). In most cases, according to this analysis, extramural research is more cost-effective than intramural research when observing uncontrolled regressions. (B–D) Linear regression results of the cost efficiency of research output measures against project types (intramural vs. extramural). The regression model was fitted for each year of the project’s progression. Unlike panel (A), this regression model controls for grant, investigator, and collaboration characteristics in order to obtain a more accurate estimate of the relative cost efficiency of intramural vs. extramural projects. The Y-axis coefficient indicates the mean disparity in research output between intramural and extramural projects, controlling for these other variables (see Methods). Because there might be covariates that could confound the data, separate regressions were conducted for all projects (B, the default), and for balanced projects using 1:1 propensity score matching (1 extramural grant for every 1 intramural grant) in order to compare grants that were the most similar to reduce the influence of unobserved covariates (C) and (D) similarly to (C) 1:4 propensity matching as a robustness check. Data and code are available at Dryad (Zheng et al., 2026).
Figure 5
Comparison of scores for human-focused research, animal research, and molecular/cellular research for intramural and extramural projects.
(A–C) These represent the average Human, Animal, and Molecular/Cellular scores for publications funded by extramural vs. intramural grants (98,648 extramural and 1594 intramural), respectively, which were downloaded from iCite (Hutchins et al., 2019b; Hutchins and Santangelo, 2019c). (A) Average scores for human-focused research for intramural research (green) and extramural research (blue) for projects of different durations: 1–3 years (left), 4–6 years (middle), and 7–10 years (right). (B) Average scores for animal research. (C) Average scores for molecular/cellular research. Mann–Whitney U tests were conducted to test the difference between the scores for intramural and extramural projects. ***p < 0.001, **p < 0.01, *p < 0.05. No asterisk indicates the difference is not statistically significant. Error bars, SEM. Data and code are available at Dryad (Zheng et al., 2026).
Appendix 1—figure 1
T-SNE plot illustrating the distribution of topic clusters, with colors consistent with those in Figure 1.
This shows a dimensionality reduction of the locations of these grants in Word2Vec space, indicating their relative proximity to one another. Grants with similar semantic meaning appear closer to one another, while those with less semantic similarity will appear farther away. Similar grants tend to cluster together in T-SNE visualizations.
Appendix 1—figure 2
T-SNE plot illustrating the distribution of grants in NIH institutes, using the same methodology as Appendix 1—figure 1.
This shows a dimensionality reduction of the locations of these grants in Word2Vec space, indicating their relative proximity to one another. Grants with similar semantic meaning appear closer to one another, while those with less semantic similarity will appear farther away. Similar grants tend to cluster together in T-SNE visualizations.
Appendix 1—figure 3
Cost breakdown for our sample of projects (see Methods) by Extramural or Intramural origin.
Appendix 1—figure 4
Cost effectiveness of intramural and extramural projects, with indirect costs for extramural projects inflated by 30%.
(A) A measure of cost effectiveness vs. progression (i.e., year of grant) for intramural research (green) and extramural research (red), for projects of different durations: 1–3 years (top row), 4–6 years (middle), and 7–10 years (bottom). These regressions do not control for other characteristics, but rather represent the raw ratios. For the first column, the Y-axis displays log10(ratio) + 1, where ratio is the cumulative total costs to the cumulative total research output for each metric (cost:output, for the first column output = #papers); error bars denote the 95% confidence intervals. The remaining columns show measures of cost effectiveness for Relative Citation Ratio, Approximate Potential to Translate, total clinical citation counts, and a binary measure of clinical citations. To account for the fact that many papers are published after funding for the relevant grant has ended, grant amounts were multiplied by a deflator – this represents the proportion of papers published to date against the anticipated number of future publications, as determined by empirical measurements (Appendix 1—table 1). In most cases, according to this analysis, extramural research is more cost-effective than intramural research when observing uncontrolled regressions. (B–D) Linear regression results of the cost efficiency of research output measures against project types (intramural vs. extramural). The regression model was fitted for each year of the project’s progression. Unlike panel (A), this regression model controls for grant, investigator, and collaboration characteristics in order to obtain a more accurate estimate of the relative cost efficiency of intramural vs. extramural projects. The Y-axis coefficient indicates the mean disparity in research output between intramural and extramural projects, controlling for these other variables (see Methods). Because there might be covariates that could confound the data, separate regressions were conducted for all projects (B, the default), and for balanced projects using 1:1 propensity score matching (1 extramural grant for every 1 intramural grant) in order to compare grants that were the most similar to reduce the influence of unobserved covariates (C) and (D) similarly to (C) 1:4 propensity matching as a robustness check.
Appendix 1—figure 5
Ternary contour plots representing the clinical citation efficiency in the human, animal, and molecular/cellular score system for intramural (D) and extramural projects (E) (Hutchins et al., 2019b; Weber, 2013).
Here, efficiency was the percentile of the cost per output in descending order. Each contour line denotes a constant efficiency percentile. Yellow/green are high-efficiency areas of the triangle, and blues are low-efficiency areas.
Appendix 1—figure 6
Distribution of papers acknowledging more than one funding source, divided, extramural and intramural.
Tables
Key resources table
| Reagent type (species) or resource | Designation | Source or reference | Identifiers | Additional information |
|---|---|---|---|---|
| Software, algorithm | Python programming language | Python | RRID:SCR_008394 |
Appendix 1—table 1
Number and proportion of papers published after grant ended by activity code prefix.
| Activity code prefix | # Papers after funding ended | # Total papers | % Funded papers published after grant ended |
|---|---|---|---|
| D | 9213 | 13,039 | 70.66 |
| F | 18,649 | 32,097 | 58.10 |
| G | 283 | 1095 | 25.84 |
| K | 43,633 | 102,892 | 42.41 |
| P | 20,766 | 50,650 | 41.00 |
| R | 225,490 | 418,989 | 53.82 |
| S | 1061 | 2176 | 48.76 |
| T | 5324 | 12,839 | 41.47 |
| U | 49,222 | 112,459 | 43.77 |
| Z | 1794 | 19,871 | 9.03 |
Additional files
Download links
A two-part list of links to download the article, or parts of the article, in various formats.
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
Comparing the outputs of intramural and extramural grants funded by National Institutes of Health
eLife 14:RP108929.
https://doi.org/10.7554/eLife.108929.3
