Meta-Research: How significant are the public dimensions of faculty work in review, promotion and tenure documents?
- Simon Fraser University, Canada
- Arizona State University, United States
- University of Vermont, United States
- Universidad Nacional Autónoma de México, Mexico
Figures
Figure 1
Percentage of institutions mentioning public and community terms and concepts by type of institution.
Bars represent whether each term or concept (several terms and phrases) was identified within documents from doctoral/research-focused universities (R-type; blue), master’s colleges and universities (M-type; orange) and baccalaureate colleges (B-type; green). The terms "public" and "community," and the concept of "public and/or community engagement" appear less often in documents from B-type institutions than from M- and R-type. The conditions of the chi-square test were not met for the term "community," but the chi-square analysis reveals the difference in presence of term "public" and concept "public and/or community engagement" are significant. Chi-square tests: Term Public: χ2 (2, N=129)=13.85, p<0.001; Concept Public and/or community engagement: χ2 (2, N=129)=9.61, p<0.01.
Figure 2
Percentage of institutions mentioning public and community terms and concepts by institution sub-type.
Bars represent whether each term or concept (several terms and phrases) was identified within documents of doctoral/research-focused universities, from the most research intensive (R1; blue), to those that are less so (R2; orange, and R3; green), as well as the Canadian research universities (RCan; red). The terms "public" and the concept of "public and/or community engagement" appear more at R1 and R2 institutions than R3, with RCan universities falling in the middle. However, sample sizes violate conditions for a chi-square test to measure the significance of these differences.
Figure 3
Percentage of institutions mentioning public and community terms and concepts by discipline.
Bars represent whether each term or concept (several terms and phrases) was identified within documents of academic units from Social Sciences and Humanities (SSH; blue), Physical Sciences and Mathematics (PSM; orange), Life Sciences (LS; green) and multidisciplinary units (red). The terms and concepts appear more frequently in LS units than others. Sample size conditions for a chi-square test were only met for the concept of "public and/or community engagement," where it indicates that the difference in this category is significant. Chi-square test: Concept Public and/or community engagement: χ2 (3, N=116)=12.45, p<0.05.
Figure 4
Relative frequency of words surrounding the term "public".
Visual representation of the relative frequency of words near (within 15 words) the word "public" across all documents. The most frequent word near "public" is "service." Along with other frequent words, this suggests that in the context of RPT, "public" is most often associated with a service activity.
Figure 5
Relative frequency of words surrounding the term "community".
Visual representation of the relative frequency of words near (within 15 words) the word "community" across all documents. The most frequent word near "community" is "university." Along with other frequent words, this suggests that the community most often referred to is that of other academics.
Figure 6
Percentage of institutions mentioning terms and concepts related to research and metrics by institution type.
Bars represent whether each term or concept (several terms and phrases) was identified within documents from doctoral/research-focused universities (R-type; blue), master’s colleges and universities (M-type; orange), and baccalaureate colleges (B-type; green). Chi-square analysis suggests that the term "impact" and the concept of "metrics" is more present at R-type than at M-type, and more present at M-type than B-type. The concept of "traditional outputs" is present at over 90% of each type, although the conditions for a chi-square test were not met for this concept or for the term "open access." Chi-square tests: Term Impact: χ2 (2, N=129)=24.13, p<0.001; Concept Metrics: χ2 (2, N=129)=32.04, p<0.001.
Figure 7
Percentage of institutions mentioning terms and concepts related to research and metrics by institution sub-type.
Bars represent whether each term or concept (several terms and phrases) was identified within documents of doctoral/research-focused universities, from the most research intensive (R1; blue), to those that are less so (R2; orange, and R3; green), as well as the Canadian research universities (RCan; red). The term "impact" appears less in R3 institutions, and the concept of "metrics" appears to decrease with research intensity (with RCan institutions at similar levels to the R2 institutions from the US) However, the conditions for a chi-square test were not met to measure the significance of these differences.
Figure 8
Percentage of institutions mentioning terms and concepts related to research and metrics by discipline.
Bars represent whether each term or concept (several terms and phrases) was identified within documents of academic units from Social Sciences and Humanities (SSH; blue), Physical Sciences and Mathematics (PSM; orange), Life Sciences (LS; green) and multidisciplinary units (red). The concept of traditional outputs is present in the vast majority of units. The term impact is more present in LS, but a chi-square test suggests the difference is not significant. The chi-square analysis also indicates the difference in the presence of the concept of "metrics" (with PSM units mentioning it the most) is significant. The conditions for a chi-square test were not met for other terms and concepts. Chi-square tests: Term Impact: χ2 (3, N=116)=5.75, p>0.05 (not significant). Concept: Metrics: χ2 (3, N=116)=7.33, p<0.05.
Figure 9
Relative frequency of words surrounding the term "impact".
Visual representation of the relative frequency of words near (within 15 words) the word "impact" across all documents. The most frequent word near "impact" is "research." Along with other frequent words, this suggests that the type of impact most valued is that which relates to research activities.
Tables
Table 1
Sampling summary of universities from Canada and the United States.
Overview of population of universities from the United States and Canada by type and sub-type, the number and percent randomly chosen for the stratified sample, and the number of institutions for which at least one relevant document was obtained.
| Number in category | Number sampled | Percent sampled | Number with documents | ||
|---|---|---|---|---|---|
| R-type | R1 | 115 | 17 | 15% | 15 |
| R2 | 107 | 16 | 15% | 15 | |
| R3 | 113 | 17 | 15% | 14 | |
| RCan | 15 | 15 | 100% | 12 | |
| M-type | M1 | 393 | 17 | 4% | 11 |
| M2 | 298 | 12 | 4% | 10 | |
| M3 | 141 | 6 | 4% | 4 | |
| MCan | 15 | 15 | 100% | 13 | |
| B-type | Bas | 259 | 14 | 5% | 11 |
| Bd | 324 | 17 | 5% | 5 | |
| BCan | 19 | 19 | 100% | 17 |
Additional files
-
Transparent reporting form
- https://doi.org/10.7554/eLife.42254.012
Download links
A two-part list of links to download the article, or parts of the article, in various formats.
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
Meta-Research: How significant are the public dimensions of faculty work in review, promotion and tenure documents?
eLife 8:e42254.
https://doi.org/10.7554/eLife.42254
