Download icon

Meta-Research: How significant are the public dimensions of faculty work in review, promotion and tenure documents?

  1. Juan P Alperin  Is a corresponding author
  2. Carol Muñoz Nieves
  3. Lesley A Schimanski
  4. Gustavo E Fischman
  5. Meredith T Niles
  6. Erin C McKiernan
  1. Simon Fraser University, Canada
  2. Arizona State University, United States
  3. University of Vermont, United States
  4. Universidad Nacional Autónoma de México, Mexico
Feature Article
Cite this article as: eLife 2019;8:e42254 doi: 10.7554/eLife.42254
9 figures, 1 table, 1 data set and 1 additional file

Figures

Percentage of institutions mentioning public and community terms and concepts by type of institution.

Bars represent whether each term or concept (several terms and phrases) was identified within documents from doctoral/research-focused universities (R-type; blue), master’s colleges and universities (M-type; orange) and baccalaureate colleges (B-type; green). The terms "public" and "community," and the concept of "public and/or community engagement" appear less often in documents from B-type institutions than from M- and R-type. The conditions of the chi-square test were not met for the term "community," but the chi-square analysis reveals the difference in presence of term "public" and concept "public and/or community engagement" are significant. Chi-square tests: Term Public: χ2 (2, N=129)=13.85, p<0.001; Concept Public and/or community engagement: χ2 (2, N=129)=9.61, p<0.01.

https://doi.org/10.7554/eLife.42254.003
Percentage of institutions mentioning public and community terms and concepts by institution sub-type.

Bars represent whether each term or concept (several terms and phrases) was identified within documents of doctoral/research-focused universities, from the most research intensive (R1; blue), to those that are less so (R2; orange, and R3; green), as well as the Canadian research universities (RCan; red). The terms "public" and the concept of "public and/or community engagement" appear more at R1 and R2 institutions than R3, with RCan universities falling in the middle. However, sample sizes violate conditions for a chi-square test to measure the significance of these differences.

https://doi.org/10.7554/eLife.42254.004
Percentage of institutions mentioning public and community terms and concepts by discipline.

Bars represent whether each term or concept (several terms and phrases) was identified within documents of academic units from Social Sciences and Humanities (SSH; blue), Physical Sciences and Mathematics (PSM; orange), Life Sciences (LS; green) and multidisciplinary units (red). The terms and concepts appear more frequently in LS units than others. Sample size conditions for a chi-square test were only met for the concept of "public and/or community engagement," where it indicates that the difference in this category is significant. Chi-square test: Concept Public and/or community engagement: χ2 (3, N=116)=12.45, p<0.05.

https://doi.org/10.7554/eLife.42254.005
Relative frequency of words surrounding the term "public".

Visual representation of the relative frequency of words near (within 15 words) the word "public" across all documents. The most frequent word near "public" is "service." Along with other frequent words, this suggests that in the context of RPT, "public" is most often associated with a service activity.

https://doi.org/10.7554/eLife.42254.006
Relative frequency of words surrounding the term "community".

Visual representation of the relative frequency of words near (within 15 words) the word "community" across all documents. The most frequent word near "community" is "university." Along with other frequent words, this suggests that the community most often referred to is that of other academics.

https://doi.org/10.7554/eLife.42254.007
Percentage of institutions mentioning terms and concepts related to research and metrics by institution type.

Bars represent whether each term or concept (several terms and phrases) was identified within documents from doctoral/research-focused universities (R-type; blue), master’s colleges and universities (M-type; orange), and baccalaureate colleges (B-type; green). Chi-square analysis suggests that the term "impact" and the concept of "metrics" is more present at R-type than at M-type, and more present at M-type than B-type. The concept of "traditional outputs" is present at over 90% of each type, although the conditions for a chi-square test were not met for this concept or for the term "open access." Chi-square tests: Term Impact: χ2 (2, N=129)=24.13, p<0.001; Concept Metrics: χ2 (2, N=129)=32.04, p<0.001.

https://doi.org/10.7554/eLife.42254.008
Percentage of institutions mentioning terms and concepts related to research and metrics by institution sub-type.

Bars represent whether each term or concept (several terms and phrases) was identified within documents of doctoral/research-focused universities, from the most research intensive (R1; blue), to those that are less so (R2; orange, and R3; green), as well as the Canadian research universities (RCan; red). The term "impact" appears less in R3 institutions, and the concept of "metrics" appears to decrease with research intensity (with RCan institutions at similar levels to the R2 institutions from the US) However, the conditions for a chi-square test were not met to measure the significance of these differences.

https://doi.org/10.7554/eLife.42254.009
Percentage of institutions mentioning terms and concepts related to research and metrics by discipline.

Bars represent whether each term or concept (several terms and phrases) was identified within documents of academic units from Social Sciences and Humanities (SSH; blue), Physical Sciences and Mathematics (PSM; orange), Life Sciences (LS; green) and multidisciplinary units (red). The concept of traditional outputs is present in the vast majority of units. The term impact is more present in LS, but a chi-square test suggests the difference is not significant. The chi-square analysis also indicates the difference in the presence of the concept of "metrics" (with PSM units mentioning it the most) is significant. The conditions for a chi-square test were not met for other terms and concepts. Chi-square tests: Term Impact: χ2 (3, N=116)=5.75, p>0.05 (not significant). Concept: Metrics: χ2 (3, N=116)=7.33, p<0.05.

https://doi.org/10.7554/eLife.42254.010
Relative frequency of words surrounding the term "impact".

Visual representation of the relative frequency of words near (within 15 words) the word "impact" across all documents. The most frequent word near "impact" is "research." Along with other frequent words, this suggests that the type of impact most valued is that which relates to research activities.

https://doi.org/10.7554/eLife.42254.011

Tables

Table 1
Sampling summary of universities from Canada and the United States.

Overview of population of universities from the United States and Canada by type and sub-type, the number and percent randomly chosen for the stratified sample, and the number of institutions for which at least one relevant document was obtained.

https://doi.org/10.7554/eLife.42254.002
Number in categoryNumber sampledPercent sampledNumber with documents
R-typeR11151715%15
R21071615%15
R31131715%14
RCan1515100%12
M-typeM1393174%11
M2298124%10
M314164%4
MCan1515100%13
B-typeBas259145%11
Bd324175%5
BCan1919100%17

Data availability

The data that support the findings of this study are available in the Harvard Dataverse with the identifier https://doi.org/10.7910/DVN/VY4TJE (Alperin et al., 2019). These data include the list of institutions and academic units for which we have acquired documents along with an indicator of whether each term and concept studied was found in the documents for the institution or academic unit. The data also include the aggregated values and chi-square calculations reported. The code used for computing these aggregations can be found on Github https://github.com/ScholCommLab/rpt-project (Alperin, 2019; copy archived at https://github.com/elifesciences-publications/rpt-project). The documents collected are available on request from the corresponding author (JPA). These documents are not publicly available due to copyright restrictions.

The following data sets were generated
  1. 1
    Harvard Dataverse
    1. JP Alperin
    2. Nieves C Muñoz
    (2018)
    Terms and Concepts found in Tenure and Promotion Guidelines from the US and Canada.
    https://doi.org/10.7910/DVN/VY4TJE

Additional files

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Download citations (links to download the citations from this article in formats compatible with various reference manager tools)

Open citations (links to open the citations from this article in various online reference manager services)