Review, promotion and tenure (RPT) processes are a cornerstone of academic life at higher education institutions in the United States and Canada. They can influence where faculty focus their attention, the activities they choose to pursue, and choices such as the direction of their research program and the venues where they publish their work, especially during the pre-tenure period (Harley et al., 2010). Unsurprisingly, RPT has been the subject of much scrutiny (for examples see Gordon, 2008; Schimanski and Alperin, 2018). While previous studies (Gardner and Veliz, 2014; Youn and Price, 2009) have documented how expectations of faculty have expanded from having to excel in either teaching, research or service, to having to demonstrate excellence in all three, research continues to be the most highly valued aspect of faculty work (Acker and Webber, 2016; Green and Baskind, 2007; Macfarlane, 2007). Teaching is typically valued less than research, despite teaching duties often representing more than half of the workload (Diamond and Adam, 1998), and service activities come a distant third (Fischman et al., 2018; Foos et al., 2004).

Where, then, in this context of ever-expanding responsibilities and emphasis on research, does a commitment to the public come into the RPT process? This depends, of course, on which concept of public one focuses on and what dimensions are emphasized. In 2010 one of us (GEF) and two colleagues offered four basic dimensions of publicness that are used in discussions about what it means for universities to fulfill their public missions (Fischman et al., 2010). Perhaps the most frequently used dimension is that which refers directly to the concept of public patronage in the sense that public universities in the United States and Canada belong to, and are administered by, federal, state or provincial agencies such as a state’s appointed board of regents. A second dimension relates to the widespread notion that public universities should be as close as possible to free of cost, or the cost should not be a barrier to access through the use of financial assistance. A third dimension of publicness stems from the belief that universities should operate with the mission of addressing general social problems, promoting the common good, and emphasizing the social contributions of educational achievement beyond the individuals’ benefit of access to higher education. Finally, the publicness of a university requires addressing the notion of accountability: to whom are higher education organizations accountable? Who represents the public interest in assessing the public effectiveness of an organization?

Notably, the work of faculty members intersects with all these dimensions: a good deal of research and development activities are supported with public money (i.e., public patronage), even at private institutions (NSF, 2016); faculty labor in the form of teaching, research and service is supposed to serve the common good and address social problems (i.e., public good), for which universities in the US and Canada receive a tax-exempt status; and, perhaps now more than ever, faculty need to demonstrate the value of their work (i.e., public accountability), and are therefore subject to more intense public scrutiny. Faculty work is also related in multiple ways to keeping the costs of access (at least at public universities) as low as possible (i.e., public access). Among other things, faculty work intersects with this economic dimension through their salaries (which are directly linked to maintaining low fees and tuition), their work as administrators, and through the expansion of their fund seeking actions (including fundraising activities not related to research grants). As universities struggle to define their own publicness, how do faculty effectively manage their careers in ways that support the various dimensions of the public mission of universities?

There appear to be organizational tensions between demands for demonstrating the public value of scholarship (i.e., public accountability) and the focus on "high prestige" or "high impact" publications by RPT committees. If publicness is interpreted as promoting public good, we might expect there to be calls for research outputs to take forms that are more ready for public consumption (not just more publicly available). Yet, determining the "prestige" of a publication venue is usually done at the discretion of evaluation committees (King et al., 2006; Seipel, 2003), through ranked lists or tiers supplied by academic institutions (Malsch and Tessier, 2015), or directly through impact factors and other citation metrics that measure use only within other scholarly works (Adler et al., 2009; Walker et al., 2010). These measures of prestige and impact reinforce the most commonly found publishing formats and venues (e.g., journal articles, books and conference presentations), which do not usually serve public needs in the way other forms do more directly (e.g., blog posts, podcasts, public outreach events). In the sense of public patronage, we might expect the emphasis on publications to move towards the use of open access (OA) models with the public gaining access to the work they are funding. OA has indeed grown, with around 50% of the most recent literature being freely available to the public (Archambault, 2018; Archambault et al., 2014; Piwowar et al., 2018). Yet, OA remains low on the priority lists of faculty (Dallmeier-Tiessen et al., 2011; Gaines, 2015; Odell et al., 2017), even when surveys indicate that many faculty believe open access to their published works is beneficial to their careers due to wider readership (Dallmeier-Tiessen et al., 2011; Gaines, 2015; Odell et al., 2017). It seems these faculty simultaneously hold the conflicting belief that traditional publishing is better for their careers overall because it is valued more in the RPT process (Migheli and Ramello, 2014; Peekhaus and Proferes, 2015; Peekhaus and Proferes, 2016; Rodriguez, 2014).

The debate about OA and of where to publish has been complemented with a growing interest for scientific measures beyond citations (so-called altmetrics; Priem et al., 2010). Some hope that these new metrics might serve as indicators of societal impact (Bornmann, 2014; Bornmann, 2015; Robinson-Garcia et al., 2017b; Konkiel, 2016). However, despite predictions that there would be a movement towards using non-citation metrics to assess the influence of research findings for RPT (Darling et al., 2013; Piwowar, 2013), there are concerns, limitations and challenges in the use of these metrics that are hampering their uptake (Gordon et al., 2015; Haustein et al., 2016; Howard, 2013; Lopez-Cozar et al., 2012). Moreover, there is little evidence that mentions on social media are correlated with citations (see Konkiel et al., 2016 for an overview) or that they can serve as indicators of public uptake (Alperin et al., 2019a; Didegah et al., 2018; Robinson-Garcia et al., 2017a).

It may be, however, that interest in developing and adopting these new metrics is not indicative of an interest in measuring the alignment between research and the public, but of growing calls for public accountability. A recent independent report commissioned by the Higher Education Funding Council for England (HEFCE) to assess the role of metrics in research assessment and management sees this as part of a "metric tide" that has been swelling in part because of “growing pressures for audit and evaluation of public spending on higher education and research” (p. 136, Wilsdon et al., 2015). Within the RPT process, there is little evidence for the inclusion of altmetrics within formal evaluation procedures (Gruzd et al., 2011; Howard, 2013), although this may be changing, as examples of altmetrics in faculty CVs have begun to emerge (cf., Webster, 2018). However, even in some documented cases where they were included (information science and medicine), department chairs did not value them towards promotion (Aharony et al., 2017; Cameron et al., 2016; Fischman et al., 2018). In contrast, there is evidence that institutions consider citation counts in their RPT process, which, by design, only measure uptake and use of the research by the academic community (Dagenais Brown, 2014; Harley et al., 2010; Reinstein et al., 2011).

Another attempt to address the public dimensions of faculty work beyond publication and dissemination formats is manifested through concerted efforts to engage communities in the research process itself. Such efforts can be seen in the growing body of work about such practices (cf. a bibliography of over 600 articles on Community Engaged Scholarship; CES Partnership Resources, 2014), and in the various statements, toolkits and standards for documenting and evaluating community engaged scholarship in faculty RPT guidelines, including the Carnegie Foundation's Elective Community Engagement Classification, the Association of Public Land-Grant Universities' Task Force on the New Engagement, the Research University Engaged Scholarship Toolkit developed by The Research University Civic Engagement Network (TRUCEN), and a partnership of eight Canadian universities that developed Rewarding Community-Engaged Scholarship: Transforming University Policies and Practices. While many faculty have embraced such community engaged scholarship and related models, there is still little evidence these are valued across the academy. In particular, Harley et al. found that faculty who find ways to give back to the community and acknowledge the support of taxpayer funding, such as by participating in public education, generally receive recognition for these efforts regardless of institution type or field of study (Harley et al., 2010). However, these kinds of activities, while representing valid social contributions that can increase a university’s accountability to the public, are often not recognized formally in the RPT process (Goldstein and Bearman, 2011).

Although previous work provides a sense of how the dimensions of publicness outlined here (public patronage, public access, public good and public accountability) intersect with the RPT process (O'Meara, 2002), more empirical work is needed to understand how publicness is incentivized in faculty careers (O’Meara, 2014; O’Meara et al., 2015). To this end we set out to collect documents, including collective agreements, faculty handbooks, guidelines and forms, that describe RPT requirements for faculty at a representative set of higher education institutions in the US and Canada. We collected these documents to analyze the degree to which various terms and concepts, in particular those that relate to research outputs and assessment, are mentioned in the RPT process, and discuss how the presence of these terms may relate to different concepts of publicness in higher education.

We began our analysis with the terms "public" and "community" to understand the degree to which the public is talked about, and to gain a sense of the context surrounding their inclusion in RPT documents. We then focused on several terms and groups of terms (i.e., concepts) that intersect with the notions of publicness identified above, starting with the concept of "public and community engagement," the presence of which would be indicative of incentives to work alongside the public in ways that more closely align research to the public’s needs. Given the importance assigned to research in the RPT process (Schimanski and Alperin, 2018), we then turned our attention to terms and concepts related to research publications and their assessment, such as "open access," "publication formats" and "metrics," all of which speak to the different aspects of publicness outlined above.

Context surrounding public and community

In analyzing RPT documents for their inclusion of concepts related to the public and community, we found that 87% of institutions mention the term "community" in either the university level or academic unit guidelines, while 75% mention the term "public."

Overall, inclusion of the terms "public" and "community" is most common in research-focused (R-type) institutions (Figure 1). Within R-type institutions, we also found a trend towards greater inclusion of these terms at those institutions with the highest level of research activity (i.e., R1). All documents at the R1 level included the terms "public" and "community," while 93% of R2 institutions included both these terms and only 83% of the Canadian R-type (RCan; Figure 2). Within the academic units of R-type institutions, we found that of the disciplines examined, the Life Sciences (LS) most frequently include these terms, with 88% including "public" and 100% including "community" (Figure 3).

To better understand the context in which the terms "public" and "community" were being used, we analyzed the most frequent words surrounding each term. With these and other terms, we considered a word to be near our term of interest if it was within the 15 words preceding or following it (the length of an average sentence; The Acropolitan, 2017). The 10 most used words surrounding "public" were, in descending order of frequency: "service," "faculty," "professional," "research," "university," "activities," "teaching," "community," "work" and "academic" (Figure 4). The 10 most used words around "community" were, in descending order of frequency: "university," "service," "faculty," "professional," "academic," "research," "activities," "members," "teaching" and "member" (Figure 5).

Figure 4

Download asset Open asset

Figure 5

Download asset Open asset

The high incidence of these terms suggests that publicness features in the RPT process in some way. Although both "service" and "research" appear among the most frequent words surrounding "public" and "community," "service" is mentioned 1,170 times near "public" and 4,184 times near "community," while "research" is mentioned less than half as much (668 times near "public" and 1,671 near "community"). This, and the other frequent words in the surrounding context, are indicative of the terms "public" and "community" being more commonly associated with the service component of RPT, which is the least highly regarded of the RPT trifecta (Fischman et al., 2018; Foos et al., 2004; Harley et al., 2010).

Instances of "service" near the word "public" often included references to "public service" as a dimension or set of activities within the service category, thus explicitly separated from research. For example, guidelines of the Faculty of Arts of the University of Regina state that “The duties of a faculty member shall normally include: teaching and related duties (hereinafter "teaching"); scholarship, research, or equivalent professional duties (hereinafter "scholarship"); participation in collegial governance (hereinafter "administrative duties" and/or public service)” (University of Regina, 2017). Similarly, guidelines of the College of Education and Behavioral Sciences at the University of Northern Colorado state that “American colleges and universities have customarily examined faculty performance in the three areas of teaching, scholarship, and service, with service sometimes divided further into public service and service to the college or university” (University of Northern Colorado, 2010). While establishing this separation between (public) service and research, some documents also mandate the relatively lower importance of this and other dimensions of service in comparison to research activities. For example, the guidelines of the Department of Economics at the University of Utah manifest that “The Department's criteria that pertain to the qualification of candidates for retention, promotion, and tenure at all levels are: research, teaching, and university, professional, and public service. Research and teaching are of primary importance in evaluating the actual and potential performance of a candidate. Service is of secondary importance, but adequate performance in this area is expected of all candidates” (University of Utah, 2007).

In the case of "community," it becomes apparent by looking at its frequent proximity to words like "university," "service," "faculty," "professional" and "academic" that this term is generally used to refer to the academic community, composed primarily of faculty members (Figure 5). Again, we often see a requirement to provide service to this particular community in statements such as the following: “Distinctive service to the University and academic community would be evidenced by the candidate having made contributions of leadership and innovation involving decisions and policies that have had a major beneficial influence” (Acadia University, 2014); and “All candidates for tenure will be expected to demonstrate … that he/she has become a responsible and contributing member of the University/academic community” (Simon Fraser University, 2013).

Although these terms are also found within the context of research, as in some of the quotes above, we noted that the word "research" can appear near the words "public" and "community" without being directly relevant to the notion of public and/or community engaged research. This motivated a more refined coding strategy for this concept.

While the context surrounding the concepts of public and/or community engaged scholarship allows us to see some of the ways in which faculty are asked to align their activities with the public good, the demarcation between this form of scholarship and "traditional research" suggests that we need to look at how the latter is discussed in the RPT guidelines separately. We therefore searched for mentions of traditional research outputs (which, as indicated above, are not typically geared towards being accessed and engaged by diverse audiences without specialized training), and whether these outputs are expected to be made publicly available (through open access), what type of impact this work is expected to have (public or otherwise), and how it is evaluated. To do this, we conducted a similar analysis to that above, but with terms related to traditional research outputs, open access, impact and citation metrics, and considered their prevalence in relation to public and community terms.

Traditional research outputs

We found that guidelines for faculty often provide specifics when it comes to the types of research outputs that can be considered for tenure and promotion. This frequently takes the form of a list of outputs that are considered valuable, although these lists sometimes also explicitly mention that other forms of scholarship are welcome. For instance, guidelines of the College of Business and Economics at Boise State University manifest that “Examples of the types of evidence which demonstrate research and scholarly activity include (but are not limited to): (1) Articles in refereed journals (2) Books or research monographs (3) Chapters in books or monographs (4) Other published articles (5) Papers presented at academic conferences and/or published in proceedings (6) Published book reviews (7) Participation as a paper discussant or panel discussant at academic conferences (8) Grants and contracts for research and scholarly activities” (Boise State University, 2008). Similarly, guidelines of Memorial University of Newfoundland establish that “Factors that may be considered [as a demonstrated record of research, scholarship, or creative and professional activities] include but are not limited to: the publication of books, monographs, and contributions to edited books; papers in both refereed and non-refereed journals; scholarly presentations delivered at professional meetings; success in grant competitions; participation in panels; unpublished research including current work in progress both supported and non-supported; editorial and refereeing duties; creative works and performances; and scholarship evidenced by the candidate's depth and breadth of knowledge and general contributions to the research life and creative milieu of the University” (Memorial University of Newfoundland, 2014).

When looking for traditional outputs (i.e., books, conference proceedings, grants, journal articles, monographs and presentations), we found at least one mentioned in 90–95% of R-, M- and B-type institutions, in all R-sub-types, and in the three disciplinary categories (it was a little below 90% for the interdisciplinary academic units; Figures 6–8). Of the terms analyzed in our study, this group related to outputs was the most consistently present across institution types and disciplines. Their consistency and relative ubiquity show that if there is one thing that is certain to count towards faculty career progression, it is producing traditional academic outputs.

Figure 6

Download asset Open asset

Figure 7

Download asset Open asset

Figure 8

Download asset Open asset

Meanwhile, other outputs resulting from faculty work that relate to the public and/or the community are sometimes considered as a service activity. For example, traditional outputs and metrics are mentioned in the "Scholarship and Research" section of the Institute of Environmental Sustainability at Loyola University’s Tenure and Promotion Guidelines, while “publishing articles for the general public” is included within the section "Professional Contributions" (which are deemed as “all service and accomplishments not defined as research … and can contribute to the general development of the broader profession”; Loyola University, 2015). Similarly, guidelines of the Department of Psychology of the University of Guelph establish that “Normally, publication of scholarly works relevant to some aspect of the discipline of psychology will be considered. Other publications (e.g., trade books, articles in popular magazines) will be evaluated under service to the community. Where appropriate, however, these products may be referenced as knowledge mobilization activities in the dossiers related to scholarship, service or community engagement” (University of Guelph, 2012b). Like these, we found many instances where faculty are offered lists of valued outputs beyond those used for communicating within the academic community, but more often than not, these are not regarded as research activities.

Impact

We went on to examine the kind of impact that is expected of faculty in the RPT process to see if, despite the encouragement of traditional research outputs and the cautionary tone around open access, faculty are asked to have impact that goes beyond the academic community. We found that "impact" is a term of interest, with 57% of institutional documents mentioning it explicitly. Use of this term is most common in RPT documents of R-type institutions (79%; Figure 6) and, similar to "public" and "community," appears most frequently within higher-ranking R-type institutions (94% at R1, 93% at R2, and 50% of R3; RCan institutions fall in the middle at 75%; Figure 7). Related, we find similar results to "public" and "community" in that "impact" is mentioned most frequently within the documents of Life Sciences academic units of R-type institutions (85%; Figure 8).

Like with the other terms of interest, we assessed the most frequently employed words surrounding the term "impact" (within 15 words) in the RPT documents. The top ten are, in descending order of frequency: "research," "candidate," "work," "faculty," "quality," "teaching," "evidence," "field," "service" and "scholarly" (Figure 9). The term "public" is the 88^th most frequent word near "impact," while "factors" and "factor" (likely referring to the Impact Factor) rank 67^th and 204^th respectively (discussed further below in the analysis of metrics).

Figure 9

Download asset Open asset

We find a higher presence of "impact" in proximity to research related terms as compared to other RPT components. Although the associated words show that the impact of faculty work is a concern across all three areas of academic activity (research, teaching and service), "impact" is mentioned 904 times near "research" versus 392 times near "teaching" and 344 times near "service." It should be said, however, that how "impact" is defined is not always entirely clear, with several instances using non-specific descriptors, such as "major impact," "substantial impact," "demonstrable impact," "considerable impact," "significant impact," "valuable impact," "outstanding impact," "total impact," "maximum impact," "minimal impact" and various others. For example, guidelines of the University of Washington-Tacoma state that “Appointment with the title of professor of practice is made to a person who is a distinguished practitioner or distinguished academician, and who has had a major impact on a field important to the University’s teaching, research, and/or service mission” (University of Washington-Tacoma, 2017). Similarly, guidelines of the Department of Biological Sciences at Simon Fraser University manifest that “The number of publications is important, but secondary to their quality and total impact, and to the applicant's contribution to the research publications” (Simon Fraser University, 2017).

Meanwhile, the public dimension of impact, in any form, is minimally addressed. Specific mentions of this concept are rare (appearing in only 9% of the R-type institutions and 11% of the M-type), and are often non-specific about how that public impact will be determined. For example, guidelines of Carleton University establish that “Evidence appropriate to the discipline or field used to demonstrate the originality and quality of research/scholarly activity or creative work in support of an application for tenure or promotion may include, but is not limited to … other publications demonstrating a high quality of scholarship with significant public impact” (Carleton University, 2014). Similarly, guidelines of the Faculty Division of Physical Sciences at the University of Wisconsin-Madison require faculty to “List the implications of the program; its relevance to the problems of agriculture, industry, and other segments of society in the state and nation; and its potential or demonstrated impact on the public” (University of Wisconsin-Madison, 2014).

Our research shows that, while there is a relatively high incidence of the terms "public" and "community" in the representative set of RPT documents – which could be interpreted as an indicator that faculty do need to consider the nature of the publicness of their work – there are neither explicit incentives, nor clear structures of support for assessing the contributions of scholarship to the various dimensions of publicness. Conversely, the higher incidence of mentions of traditional research outputs (which are not typically easy for the public to access and require specialized knowledge to be understood), the almost non-existent mentions of open access (which would grant the public access to all research), and the persistent presence of traditional citation metrics (which do not account for public use of scholarly work) indicate that, in order to be successful, faculty are mostly incentivized towards research activities that can be counted and assessed within established academic conventions.

Moreover, our analysis found that RPT documents signal that faculty should focus on uptake within their specific academic fields, especially at the R-type institutions where quantifiable citation-based metrics are mentioned in the documents of nearly three quarters of the institutions studied. As faculty careers are more closely scrutinized through metrics that seek to reflect research use and value within academia (i.e., citations; Dahler-Larsen (2011); Fischman et al., 2018; Hicks et al., 2015; Wilsdon et al., 2015), the ability for faculty to dedicate time and energy into activities that more directly serve the public good are not incentivized.

We want to be very explicit that we do not oppose the use of well-defined indicators or metrics as one way (among many others) to assess the scholarly relevance of research. However, we suggest care is needed in identifying, and replacing, any simplistic policies that only pay lip service and symbolic attention to the public dimensions of scholarship and that inadvertently generate barriers to publicness by encouraging the use of poorly constructed metrics to assess research productivity. We are not the first to identify this need. The Humane Metrics Initiative (HuMetricsHSS; https://humetricshss.org/about/), for example, has been working towards identifying metrics that support specific values, including engaging with one’s community of practice and with the public at large. More broadly, there are many universities and individuals working on overcoming the limitations and the adverse effects that the use of metrics is producing; among these efforts, the Declaration on Research Assessment (DORA, 2018b) stands out with over 13,000 scholars and over 1,000 scholarly organizations as signatories who have expressed their commitment to avoiding simplistic models to assess scholarly impact (see also Hicks et al., 2015; O’Neill, 2016; Simons, 2008; Vanclay, 2012).

To this end, our work informs these efforts by identifying the specific modes of scholarship and assessment measures that are prevalent in current policies. We believe that our findings can help faculty reflect on how they focus their energies and characterize their efforts when they are being evaluated, while at the same time giving those conducting the evaluations (i.e., RPT committees, department chairs and deans) a greater understanding of how the guidelines at their institution may be inadvertently promoting certain forms of scholarship and assessment measures over others, which may be at odds with the public missions of many institutions. We suggest that, given the prominence of public and related terms in RPT documents and the lack of explicit metrics or incentives to encourage publicly-oriented scholarship, there are clear opportunities for institutions to reconcile these discrepancies.

Where we do find evidence for the promotion of specific forms of scholarship is in the types of outputs that are mentioned in the documents. While in this study we did not analyze all the outputs being asked of faculty, we found an almost ubiquitous presence of traditional research formats (i.e., books, conference proceedings, grants, journal articles, monographs and presentations) which are often not accessible to the public who ultimately underwrites the work. The remarkably consistent presence of these few terms across institution type and sub-type, and across disciplines, is likely not surprising to most readers, but is nonetheless a reminder of how entrenched these modes of scholarship are in academia.

What might be more surprising is the lack of positive mentions of OA as a way of facilitating the uptake of these deeply entrenched formats by a more diverse set of users through increased access. OA could be a bridge that links research activities, published in traditional formats, to expanded engagement with more diverse groups of users and stakeholders, fulfilling the public patronage imperative of universities, but it does not advance a university’s efforts along the other dimensions of publicness.

Our work thus highlights some of the ways institutions could better align metrics and incentives with the different dimensions of publicness to ensure these are adequately supported in the RPT process. It seems natural for those wanting to see change in the way public and/or community engaged scholarship is valued to want to see those changes reflected in the guidelines. Our findings show that these forms of scholarship are not always regarded as highly as “published research in top-ranked/High Impact Factor journals” and are often considered part of faculty service – the least valued aspect of the RPT trifecta. The lack of value placed on service creates disparity between faculty, something that warrants special consideration given that women spend more time on such roles, often at the expense of their career progression (Guarino and Borden, 2017; Misra et al., 2011).

Of course, there are different degrees of value placed on public and community engagement activities across universities and units. The appearance of terms related to public and/or community engaged scholarship in many of the guidelines from academic units from the Life Sciences (where medical schools are found), suggests that these forms of scholarship receive consideration in some of the fields where there are direct and obvious implications for the community, and where efforts are being made for more comprehensive models of research assessment (Cabrera et al., 2018; Cabrera et al., 2017).

Counting public and/or community engaged scholarship wholly as a research activity is just one way in which publicness could be better supported, but, as O’Meara states, “just because a college changes its written definition of scholarship in promotion policies does not mean that institutional members wake up the next day with a new view of faculty work” (p. 58, O'Meara, 2002). Other efforts, such as the requirement of the International Development Research Centre (IDRC) and other Canadian agencies for knowledge mobilization plans in all of their grants, are trying to promote publicness through funding incentives (Lebel and McLean, 2018). Others still have suggested expanding the RPT trifecta to introduce a new category that includes activities that aim to disseminate information to a broader public, and that might be seen as a midpoint between research and service (Harley et al., 2010; Scheinfeldt, 2008). However, there may be limits to what additional categories related to "publicness" can achieve – particularly if they are not based on well-defined metrics – without understanding the limitations of the current assessment practices. Such categories may end up being undervalued in much the same way service is today.

Instead, our research confirms a discrepancy between how faculty work is assessed and incentivized through the RPT process and the stated goals of institutions to achieve scholarship for the public good. However, previous efforts at RPT reform suggest that solely changing what is written in RPT documents may not be sufficient to better align assessment practices and institutional goals (O’Meara, 2005). To close this gap, publicly orientated faculty work may first need to be considered on par with activities for which there are "quantifiable research metrics," since these are the ones that appear to be the most valued. That is, it seems difficult for faculty to carry out scholarly work aligned with the public dimensions of universities if this work is an additional burden that is separate from the main activity of producing knowledge.

Moreover, to close the gap, faculty may need to be allowed and likely encouraged to produce other types of outputs beyond the six traditional outputs we searched for. Relatedly, for the public availability of these and other outputs to be valued, that too may need to be explicitly rewarded. Such a change would help incorporate other forms of scholarship (e.g., software and data) and publicly oriented outputs (e.g., blog posts, policy briefs, podcasts), while, at the same time, promoting open access to all faculty work. Lastly, as mentioned above, a shift towards a more nuanced and judicious use of research metrics may allow for a greater number of activities, including those that are not readily quantifiable, to be considered and valued. Such a change is encouraged by the first of the principles in the Leiden Manifesto that states that “quantitative evaluation should support qualitative, expert assessment,” not supplant it (Hicks et al., 2015).

While these suggestions may not fully align an institution’s simultaneous goals of public good and academic productivity and output, we do believe that changing the guidelines and procedures governing the RPT process can have a significant impact on how faculty choose to allocate their time and energy. At the risk of putting too much emphasis on one initiative (for a review of different initiatives, see Moher et al., 2018), we once again can point to the efforts of DORA in identifying good practices found in the documents of several research organizations (DORA, 2018a). However, for these and any other efforts to have the intended effect, more work continues to be needed to understand the relationship between RPT guidelines and faculty behavior (calls for such work go back as early as O'Meara, 2002).

The question therefore remains: do RPT guidelines truly influence faculty priorities and publishing strategies? Our analysis offers a glimpse of the extent to which various aspects of faculty work are present in formal guidelines, but it cannot tell us whether the presence of these terms, or the way they are used, is actually affecting how faculty spend their time, nor the successes and challenges they are finding through each activity. We believe further qualitative analysis of the sample of documents we collected, combined with surveys and interviews with faculty and RPT committees, could serve to explore the relationship that these documents have with the lived experience of RPT and to further understand how publicness intersects with faculty work. In the meantime, our work leads us to confirm that faculty are more often rewarded for publishing traditional research outputs and demonstrating that those outputs are cited by other scholars than for truly promoting public scholarship. As such, there is great potential to better align public scholarship goals with the metrics and RPT process that guides faculty work.

The data that support the findings of this study are available in the Harvard Dataverse with the identifier https://doi.org/10.7910/DVN/VY4TJE (Alperin et al., 2019). These data include the list of institutions and academic units for which we have acquired documents along with an indicator of whether each term and concept studied was found in the documents for the institution or academic unit. The data also include the aggregated values and chi-square calculations reported. The code used for computing these aggregations can be found on Github https://github.com/ScholCommLab/rpt-project (Alperin, 2019; copy archived at https://github.com/elifesciences-publications/rpt-project). The documents collected are available on request from the corresponding author (JPA). These documents are not publicly available due to copyright restrictions.

1. 1. Alperin JP
   2. Muñoz Nieves C

   (2018) Harvard Dataverse

   Terms and Concepts found in Tenure and Promotion Guidelines from the US and Canada.

   https://doi.org/10.7910/DVN/VY4TJE

1. Report

   1. Acadia University

   (2014)

   Collective Agreement Between the Board of Governors and the Faculty Association

   Acadia University.

   • Google Scholar
2. 1. Acker S
   2. Webber M

   (2016)

   Discipline and Publish: The Tenure Review Process in Ontario Universities

   233–255, Assembling and Governing the Higher Education Institution, Discipline and Publish: The Tenure Review Process in Ontario Universities, London, Palgrave Macmillan.

   • Google Scholar
3. 1. Adler R
   2. Ewing J
   3. Taylor P

   (2009) Citation statistics

   Statistical Science 24:1–14.

   https://doi.org/10.1214/09-STS285

   • Google Scholar
4. 1. Aharony N
   2. Bar-Ilan J
   3. Julien H
   4. Benyamin-Kahana M
   5. Cooper T

   (2017) Acceptance of altmetrics by LIS scholars: an exploratory study

   Journal of Librarianship and Information Science 1:096100061774246.

   https://doi.org/10.1177/0961000617742461

   • Google Scholar
5. Data

   1. Alperin JP
   2. Muñoz Nieves C
   3. Schimanski L
   4. McKiernan EC

   (authors) (2018) Terms and concepts found in tenure and promotion guidelines from the US and Canada

   Harvard Dataverse.

   https://doi.org/10.7910/DVN/VY4TJE
6. 1. Alperin JP
   2. Gomez CJ
   3. Haustein S

   (2019a) Identifying diffusion patterns of research articles on Twitter: A case study of online engagement with open access articles

   Public Understanding of Science 28:2–18.

   https://doi.org/10.1177/0963662518761733

   • PubMed
   • Google Scholar
7. Software

   1. Alperin JP

   (2019b) Code associated with the review, promotion and tenure project: scholcommlab/rpt-project, version 4026daa

   GitHub.

   https://github.com/ScholCommLab/rpt-project
8. Report

   1. Archambault E
   2. Amyot D
   3. Deschamps P
   4. Nicol A
   5. Provencher F
   6. Rebout L
   7. Roberge G

   (2014)

   Proportion of Open Access Papers Published in Peer-Reviewed Journals at the European and World Levels—1996–2013

   University of Nebraska.

   • Google Scholar
9. Conference

   1. Archambault E

   (2018)

   Universalisation of scientifc dissemination presentation

   Scielo 20 Years Conference.

   • Google Scholar
10. Report

    1. Barreno L
    2. Elliott PW
    3. Madueke I
    4. Sarny D

    (2013) Community Engaged Scholarship and Faculty Assessment: A Review of Canadian Practices

    University of Saskatchewan.

    http://engagedscholarship.ca/wp-content/uploads/2014/10/Faculty-Assesment-Paper1.pdf

    • Google Scholar
11. Website

    1. BOAI

    (2002) Read the Budapest Open Access Initiative

    Accessed September 17, 2018.

    https://www.budapestopenaccessinitiative.org/read
12. Report

    1. Boise State University

    (2008)

    College of Business and Economics: Promotion and Tenure Guidelines

    Boise State University.

    • Google Scholar
13. 1. Bornmann L

    (2014) Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics

    Journal of Informetrics 8:895–903.

    https://doi.org/10.1016/j.joi.2014.09.005

    • Google Scholar
14. 1. Bornmann L

    (2015) Usefulness of altmetrics for measuring the broader impact of research: A case study using data from PLOS and F1000Prime

    Aslib Journal of Information Management 67:305–319.

    https://doi.org/10.1108/AJIM-09-2014-0115

    • Google Scholar
15. 1. Cabrera D
    2. Vartabedian BS
    3. Spinner RJ
    4. Jordan BL
    5. Aase LA
    6. Timimi FK

    (2017) More than likes and tweets: Creating social media portfolios for academic promotion and tenure

    Journal of Graduate Medical Education 9:421–425.

    https://doi.org/10.4300/JGME-D-17-00171.1

    • PubMed
    • Google Scholar
16. 1. Cabrera D
    2. Roy D
    3. Chisolm MS

    (2018) Social media scholarship and alternative metrics for academic promotion and Tenure

    Journal of the American College of Radiology 15:135–141.

    https://doi.org/10.1016/j.jacr.2017.09.012

    • PubMed
    • Google Scholar
17. 1. Cameron CB
    2. Nair V
    3. Varma M
    4. Adams M
    5. Jhaveri KD
    6. Sparks MA

    (2016) Does academic blogging enhance promotion and tenure? A survey of US and canadian medicine and pediatric department chairs

    JMIR Medical Education 2:e10.

    https://doi.org/10.2196/mededu.4867

    • PubMed
    • Google Scholar
18. Report

    1. Carleton University

    (2014)

    Collective Agreement

    Carleton University.

    • Google Scholar
19. Website

    1. CES Partnership Resources

    (2014) CES partnership resources

    Accessed January 10, 2019.

    https://www.zotero.org/groups/78525/ces_partnership_resources
20. 1. Dagenais Brown J

    (2014) Citation searching for tenure and promotion: an overview of issues and tools

    Reference Services Review 42:70–89.

    https://doi.org/10.1108/RSR-05-2013-0023

    • Google Scholar
21. Book

    1. Dahler-Larsen P

    (2011)

    The Evaluation Society

    Stanford: Stanford University Press.

    • Google Scholar
22. Preprint

    1. Dallmeier-Tiessen S
    2. Darby R
    3. Goerner B
    4. Hyppoelae J
    5. Igo-Kemenes P
    6. Kahn D
    7. Lengenfelder A

    (2011) Highlights from the SOAP project survey

    arXiv.

    https://arxiv.org/pdf/1101.5260.pdf

    • Google Scholar
23. Preprint

    1. Darling ES
    2. Shiffman D
    3. Côté IM
    4. Drew JA

    (2013) The role of twitter in the life cycle of a scientific publication

    arXiv.

    http://arxiv.org/abs/1305.0435

    • Google Scholar
24. Book

    1. Diamond RM
    2. Adam BE

    (1998)

    Changing Priorities at Research Universities

    Syracuse University.

    • Google Scholar
25. 1. Didegah F
    2. Bowman TD
    3. Holmberg K

    (2018) On the differences between citations and altmetrics: An investigation of factors driving altmetrics versus citations for Finnish articles

    Journal of the Association for Information Science and Technology 69:832–843.

    https://doi.org/10.1002/asi.23934

    • Google Scholar
26. Website

    1. DORA

    (2018a) Good practices – Funders

    Accessed September 17, 2018.

    https://sfdora.org/good-practices/funders/
27. Website

    1. DORA

    (2018b) Signers – DORA

    Accessed February 24, 2018.

    https://sfdora.org/signers/
28. 1. Fischman GE
    2. Igo S
    3. Rhoten D

    (2010)

    Global Transformation in Public Research Universities

    Great expectations, past promises, and golden ages: Rethinking the “crisis” of public research universities, Global Transformation in Public Research Universities, New York, US, Social Science Research Council.

    • Google Scholar
29. 1. Fischman GE
    2. Anderson KT
    3. Tefera AA
    4. Zuiker SJ

    (2018) If mobilizing educational research is the answer, who can afford to ask the question? An analysis of faculty perspectives on knowledge mobilization for scholarship in education

    AERA Open 4:233285841775013.

    https://doi.org/10.1177/2332858417750133

    • Google Scholar
30. Report

    1. Foos A
    2. Holmes MA
    3. O’Connell S

    (2004)

    What Does It Take to Get Tenure? Papers in the Geosciences, Paper 88

    University of Nebraska.

    • Google Scholar
31. 1. Gaines AM

    (2015) From concerned to cautiously optimistic: Assessing faculty perceptions and knowledge of open access in a campus-wide study

    Journal of Librarianship and Scholarly Communication 3:.

    https://doi.org/10.7710/2162-3309.1212

    • Google Scholar
32. 1. Gardner SK
    2. Veliz D

    (2014) Evincing the ratchet: a thematic analysis of the promotion and tenure guidelines at a striving university

    The Review of Higher Education 38:105–132.

    https://doi.org/10.1353/rhe.2014.0045

    • Google Scholar
33. Report

    1. Georgetown University

    (2017)

    Guidelines for Submission of Rank and Tenure Applications by Tenure-Line Faculty

    Georgetown University.

    • Google Scholar
34. 1. Goldstein AO
    2. Bearman RS

    (2011) Community engagement in US and Canadian medical schools

    Advances in Medical Education and Practice 2:43–49.

    https://doi.org/10.2147/AMEP.S16823

    • PubMed
    • Google Scholar
35. Book

    1. Gordon CK

    (2008)

    Organizational Rhetoric in the Academy: Junior Faculty Perceptions and Roles

    University of North Texas.

    • Google Scholar
36. 1. Gordon G
    2. Lin J
    3. Cave R
    4. Dandrea R

    (2015) The question of data integrity in article-level metrics

    PLOS Biology 13:e1002161.

    https://doi.org/10.1371/journal.pbio.1002161

    • PubMed
    • Google Scholar
37. 1. Green RG
    2. Baskind FR

    (2007) The second decade of the faculty publication project: Journal article publications and the importance of faculty scholarship

    Journal of Social Work Education 43:281–296.

    https://doi.org/10.5175/JSWE.2007.200600050

    • Google Scholar
38. 1. Gruzd A
    2. Staves K
    3. Wilk A

    (2011) Tenure and promotion in the age of online social media

    Proceedings of the American Society for Information Science and Technology 48:1–9.

    https://doi.org/10.1002/meet.2011.14504801154

    • Google Scholar
39. 1. Guarino CM
    2. Borden VMH

    (2017) Faculty service loads and gender: Are women taking care of the academic family?

    Research in Higher Education 58:672–694.

    https://doi.org/10.1007/s11162-017-9454-2

    • Google Scholar
40. Report

    1. Harley D
    2. Acord SK
    3. Earl-Novell S
    4. Lawrence S
    5. King CJ

    (2010) Assessing the Future Landscape of Scholarly Communication: An Exploration of Faculty Values and Needs in Seven Disciplines

    Center for Studies in Higher Education.

    http://escholarship.org/uc/item/15x7385g

    • Google Scholar
41. 1. Haustein S
    2. Bowman TD
    3. Holmberg K
    4. Tsou A
    5. Sugimoto CR
    6. Larivière V

    (2016) Tweets as impact indicators: Examining the implications of automated “bot” accounts on Twitter

    Journal of the Association for Information Science and Technology 67:232–238.

    https://doi.org/10.1002/asi.23456

    • Google Scholar
42. 1. Hicks D
    2. Wouters P
    3. Waltman L
    4. de Rijcke S
    5. Rafols I

    (2015) Bibliometrics: The Leiden Manifesto for research metrics

    Nature 520:429–431.

    https://doi.org/10.1038/520429a

    • Google Scholar
43. Website

    1. Howard J

    (2013) Rise of “altmetrics” revives questions about how to measure impact of research

    The Chronicle of Higher Education. Accessed June 3, 2013.

    https://www.chronicle.com/article/Rise-of-Altmetrics-Revives/139557
44. Report

    1. Kalamazoo College

    (2016)

    Faculty Handbook

    Kalamazoo College.

    • Google Scholar
45. Report

    1. King CJ
    2. Harley D
    3. Earl-Novell S
    4. Arter J
    5. Lawrence S
    6. Perciali I

    (2006) Scholarly Communication: Academic Values and Sustainable Models

    Center for Studies in Higher Education.

    http://escholarship.org/uc/item/4j89c3f7

    • Google Scholar
46. 1. Konkiel S

    (2016) Altmetrics: diversifying the understanding of influential scholarship

    Palgrave Communications 2:16057.

    https://doi.org/10.1057/palcomms.2016.57

    • Google Scholar
47. Website

    1. Konkiel S
    2. Sugimoto C
    3. Williams S

    (2016) The use of altmetrics in promotion and tenure

    Accessed September 17, 2018.

    https://er.educause.edu/articles/2016/3/the-use-of-altmetrics-in-promotion-and-tenure
48. 1. Lebel J
    2. McLean R

    (2018) A better measure of research from the global south

    Nature 559:23–26.

    https://doi.org/10.1038/d41586-018-05581-4

    • PubMed
    • Google Scholar
49. Preprint

    1. Lopez-Cozar ED
    2. Robinson-Garcia N
    3. Torres-Salinas D

    (2012) Manipulating Google Scholar citations and Google Scholar metrics: simple, easy and tempting

    arXiv.

    http://arxiv.org/abs/1212.0638

    • Google Scholar
50. Report

    1. Loyola University

    (2015)

    Institute of Environmental Sustainability: Promotion and Tenure Guidelines

    Loyola University.

    • Google Scholar
51. 1. Macfarlane B

    (2007) Defining and rewarding academic citizenship: The implications for university promotions policy

    Journal of Higher Education Policy and Management 29:261–273.

    https://doi.org/10.1080/13600800701457863

    • Google Scholar
52. 1. Malsch B
    2. Tessier S

    (2015) Journal ranking effects on junior academics: Identity fragmentation and politicization

    Critical Perspectives on Accounting 26:84–98.

    https://doi.org/10.1016/j.cpa.2014.02.006

    • Google Scholar
53. Report

    1. McGill University

    (2016)

    Tenure Regulations

     McGill University.

    • Google Scholar
54. Report

    1. Memorial University of Newfoundland

    (2014)

    Collective Agreement Between the University and the Faculty Association

    Memorial University of Newfoundland.

    • Google Scholar
55. 1. Migheli M
    2. Ramello GB

    (2014) Open access journals and academics' behavior

    Economic Inquiry 52:1250–1266.

    https://doi.org/10.1111/ecin.12131

    • Google Scholar
56. Website

    1. Misra J
    2. Lundquist JH
    3. Holmes E
    4. Agiomavritis S

    (2011) Status of women: Gender and the ivory ceiling of service work in the academy

    Accessed September 19, 2018.

    https://www.ideas-idees.ca/blog/status-women-gender-and-ivory-ceiling-service-work-academy
57. 1. Moher D
    2. Naudet F
    3. Cristea IA
    4. Miedema F
    5. Ioannidis JPA
    6. Goodman SN

    (2018) Assessing scientists for hiring, promotion, and tenure

    PLOS Biology 16:e2004089.

    https://doi.org/10.1371/journal.pbio.2004089

    • PubMed
    • Google Scholar
58. Website

    1. National Academy of Sciences

    (2006) Taxonomy of fields and their subfields

    Accessed June 13, 2017.

    http://sites.nationalacademies.org/PGA/Resdoc/PGA_044522
59. Website

    1. NSF

    (2016) NCSES higher education research and development short form: fiscal year

    Accessed September 17, 2018.

    https://ncsesdata.nsf.gov/herd/2016/html/HERD2016_DST_03.html
60. 1. O'Meara KA

    (2002) Uncovering the values in faculty evaluation of service as scholarship

    The Review of Higher Education 26:57–80.

    https://doi.org/10.1353/rhe.2002.0028

    • Google Scholar
61. 1. Odell J
    2. Palmer K
    3. Dill E

    (2017) Faculty attitudes toward open access and scholarly communications: disciplinary differences on an urban and health science campus

    Journal of Librarianship and Scholarly Communication 5:.

    https://doi.org/10.7710/2162-3309.2169

    • Google Scholar
62. 1. O’Meara KA

    (2005) Encouraging multiple forms of scholarship in faculty reward systems: Does it make a difference?

    Research in Higher Education 46:479–510.

    https://doi.org/10.1007/s11162-005-3362-6

    • Google Scholar
63. Website

    1. O’Meara K

    (2014) Change the tenure system

    Accessed March 18, 2018.

    https://www.insidehighered.com/advice/2014/01/13/essay-calls-reform-tenure-and-promotion-system
64. 1. O’Meara K
    2. Eatman T
    3. Petersen S

    (2015)

    Advancing engaged scholarship in promotion and tenure: a roadmap and call for reform

    Liberal Education 101:.

    • Google Scholar
65. Website

    1. O’Neill J

    (2016) NISO recommended practice: outputs of the alternative assessment metrics project

    Collaborative Librarianship. Accessed September 14, 2016.

    https://www.niso.org/publications/rp-25-2016-altmetrics
66. 1. Peekhaus W
    2. Proferes N

    (2015) How library and information science faculty perceive and engage with open access

    Journal of Information Science 41:640–661.

    https://doi.org/10.1177/0165551515587855

    • Google Scholar
67. 1. Peekhaus W
    2. Proferes N

    (2016) An examination of North American Library and Information Studies faculty perceptions of and experience with open-access scholarly publishing

    Library & Information Science Research 38:18–29.

    https://doi.org/10.1016/j.lisr.2016.01.003

    • Google Scholar
68. 1. Piwowar H

    (2013) Altmetrics: value all research products

    Nature 493:159.

    https://doi.org/10.1038/493159a

    • PubMed
    • Google Scholar
69. 1. Piwowar H
    2. Priem J
    3. Larivière V
    4. Alperin JP
    5. Matthias L
    6. Norlander B
    7. Farley A
    8. West J
    9. Haustein S

    (2018) The state of OA: a large-scale analysis of the prevalence and impact of open access articles

    PeerJ 6:e4375.

    https://doi.org/10.7717/peerj.4375

    • PubMed
    • Google Scholar
70. Website

    1. Priem J
    2. Taraborelli D
    3. Groth P
    4. Neylon C

    (2010) Altmetrics: a manifesto

    Accessed May 8, 2018.

    http://altmetrics.org/manifesto/
71. Report

    1. Purdue University

    (2014)

    Department of Anthropology: Promotion and Tenure Guidelines

    Purdue University.

    • Google Scholar
72. 1. Reinstein A
    2. Hasselback JR
    3. Riley ME
    4. Sinason DH

    (2011) Pitfalls of using citation indices for making academic accounting promotion, tenure, teaching load, and merit pay decisions

    Issues in Accounting Education 26:99–131.

    https://doi.org/10.2308/iace.2011.26.1.99

    • Google Scholar
73. 1. Robinson-Garcia N
    2. Costas R
    3. Isett K
    4. Melkers J
    5. Hicks D

    (2017a) The unbearable emptiness of tweeting—About journal articles

    PLOS ONE 12:e0183551.

    https://doi.org/10.1371/journal.pone.0183551

    • PubMed
    • Google Scholar
74. 1. Robinson-Garcia N
    2. van Leeuwen TN
    3. Ràfols I

    (2017b) Using altmetrics for contextualised mapping of societal impact: from hits to networks

    Social Science Research Network.

    https://doi.org/10.1093/scipol/scy024

    • Google Scholar
75. 1. Rodriguez JE

    (2014) Awareness and attitudes about open access publishing: A glance at generational differences

    The Journal of Academic Librarianship 40:604–610.

    https://doi.org/10.1016/j.acalib.2014.07.013

    • Google Scholar
76. Report

    1. San Diego State University

    (2016)

    FAQ About CAP

    San Diego State University.

    • Google Scholar
77. Website

    1. Scheinfeldt T

    (2008) Making it count: Toward a third way

    Accessed January 23, 2017.

    http://foundhistory.org/2008/10/making-it-count-toward-a-third-way/
78. 1. Schimanski L
    2. Alperin JP

    (2018) The evaluation of scholarship in the academic promotion and tenure process: Past, present, and future

    F1000Research 5:1605.

    https://doi.org/10.12688/f1000research.16493.1

    • Google Scholar
79. 1. Seipel MMO

    (2003) Assessing publication for tenure

    Journal of Social Work Education 39:79–88.

    https://doi.org/10.1080/10437797.2003.10779120

    • Google Scholar
80. Report

    1. Simon Fraser University

    (2013)

    School of Engineering Science: Articles of Governance

    Simon Fraser University.

    • Google Scholar
81. Report

    1. Simon Fraser University

    (2017)

    Department of Biological Sciences: Tenure & Promotion Criteria

    Simon Fraser University.

    • Google Scholar
82. 1. Simons K

    (2008) The misused impact factor

    Science 322:165.

    https://doi.org/10.1126/science.1165316

    • Google Scholar
83. Website

    1. The Acropolitan

    (2017) Sentence length has declined 75% in the past 500 years

    Accessed January 7, 2019.

    https://medium.com/@theacropolitan/sentence-length-has-declined-75-in-the-past-500-years-2e40f80f589f
84. Report

    1. Thomas University

    (2016)

    Faculty Handbook

    Thomas University.

    • Google Scholar
85. Report

    1. University of Calgary

    (2008)

    Faculty of Veterinary Medicine: Guidelines for Appointment, Promotion and Tenure of Academic Staff

    University of Calgary.

    • Google Scholar
86. Report

    1. University of California San Diego

    (2015)

    Committee on Academic Personnel [Memo]

    University of California San Diego.

    • Google Scholar
87. Report

    1. University of Central Florida

    (2014)

    Department of Writing and Rhetoric: Promotion and Tenure Guidelines

    University of Central Florida.

    • Google Scholar
88. Report

    1. University of Central Florida

    (2015)

    Department of Sociology: Department Criteria for Tenure and Promotion

    University of Central Florida.

    • Google Scholar
89. Report

    1. University of Guelph

    (2012a)

    Department of Political Science: Department Guidelines Document for Tenure, Promotion, and Performance Assessment

    University of Guelph.

    • Google Scholar
90. Report

    1. University of Guelph

    (2012b)

    Department of Psychology: Document for Tenure, Promotion, and Performance Assessment

    University of Guelph.

    • Google Scholar
91. Report

    1. University of North Carolina at Chapel Hill

    (2009)

    Report of the UNC Task Force on Future Promotion and Tenure Policies and Practices

    University of North Carolina at Chapel Hill.

    • Google Scholar
92. Report

    1. University of North Carolina at Chapel Hill

    (2015)

    School of Information and Library Science: Criteria and Procedures for Faculty Appointments, Reappointments, Promotion and Tenure

    University of North Carolina at Chapel Hill.

    • Google Scholar
93. Report

    1. University of Northern Colorado

    (2010)

    College of Education and Behavioral Sciences: Comprehensive Review Process and Guidelines

    University of Northern Colorado.

    • Google Scholar
94. Report

    1. University of Regina

    (2017)

    Faculty of Arts: Appendix A. Criteria for Performance Review

    University of Regina.

    • Google Scholar
95. Report

    1. University of Southern Mississippi

    (2010)

    Department of Geography and Geology: Tenure and Promotion Policies

    University of Southern Mississippi.

    • Google Scholar
96. Report

    1. University of Southern Mississippi

    (2016a)

    Department of Political Sciences: Department Guidelines for Tenure and Promotion

    University of Southern Mississippi.

    • Google Scholar
97. Report

    1. University of Southern Mississippi

    (2016b)

    Faculty Handbook

    University of Southern Mississippi.

    • Google Scholar
98. Report

    1. University of Utah

    (2000)

    Department of Anthropology: Guidelines for Retention, Promotion, and Tenure

    University of Utah.

    • Google Scholar
99. Report

    1. University of Utah

    (2007)

    Department of Economics Criteria for Retention, Promotion, and Tenure

    University of Utah.

    • Google Scholar
100. Report

     1. University of Washington-Tacoma

     (2017)

     Faculty Code. Chapter 24: Appointment and Promotion of Faculty Members

     University of Washington-Tacoma.

     • Google Scholar
101. Report

     1. University of Wisconsin-Madison

     (2014)

     Faculty Division of Physical Sciences: Guidelines for Recommendations for Promotion or Appointment to Tenure

     University of Wisconsin-Madison.

     • Google Scholar
102. Report

     1. University of Wisconsin-Madison

     (2016)

     Faculty Division of Biological Sciences: Tenure Guidelines

     University of Wisconsin-Madison.

     • Google Scholar
103. 1. Vanclay JK

     (2012) Impact factor: outdated artefact or stepping-stone to journal certification?

     Scientometrics 92:211–238.

     https://doi.org/10.1007/s11192-011-0561-0

     • Google Scholar
104. 1. Walker RL
     2. Sykes L
     3. Hemmelgarn BR
     4. Quan H

     (2010) Authors' opinions on publication in relation to annual performance assessment

     BMC Medical Education 10:21.

     https://doi.org/10.1186/1472-6920-10-21

     • PubMed
     • Google Scholar
105. Website

     1. Webster B

     (2018) LibGuides: altmetrics: use cases

     Accessed September 19, 2018.

     https://pitt.libguides.com/altmetrics/usecases
106. Report

     1. Wilsdon J
     2. Allen L
     3. Belfiore E
     4. Campbell P
     5. Curry S
     6. Hill S
     7. Jones R
     8. Kain R
     9. Kerridge S
     10. Thelwall M
     11. Tinkler J
     12. Viney I
     13. Wouters P
     14. Hill J
     15. Johnson B

     (2015)

     The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management

     HEFCE.

     • Google Scholar
107. Report

     1. Windsor University

     (2016)

     Sample Research Evaluation Rubric

     Windsor University.

     • Google Scholar
108. 1. Youn TIK
     2. Price TM

     (2009) Learning from the experience of others: the evolution of faculty tenure and promotion rules in comprehensive institutions

     The Journal of Higher Education 80:204–237.

     https://doi.org/10.1080/00221546.2009.11772139

     • Google Scholar

Thank you for submitting your article "How significant are the public dimensions of faculty work in review, promotion, and tenure documents?" for consideration by eLife. Your article has been reviewed by three peer reviewers, and the evaluation has been overseen by Emma Pewsey as the Reviewing Editor and Peter Rodgers as the Senior Editor. The following individuals involved in review of your submission have agreed to reveal their identity: Emily Janke (Reviewer #1); Heather Coates (Reviewer #2).

The reviewers have discussed the reviews with one another and I have drafted this decision to help you prepare a revised submission.

Summary:

This is an important and timely study that provides a unique perspective on how higher education interacts with, serves, and is viewed by the public. The study contributes an important data set and novel analysis techniques that contextualize previous studies, describe important characteristics and set a foundation for longitudinal analysis that could be incredibly valuable in capturing how review, promotion and tenure processes change over time. However, there are a number of concerns to be addressed regarding its analytical framework, analysis and discussion that will need to be addressed before publication.

Essential revisions:

Definition of terms

1) Please revise the text to be clear and consistent in your use of terms such as community, community engagement, public scholarship, and public engagements, and make sure these are all introduced in the Introduction section. This is important, particularly around the term “community engagement.” For example, in the subsection “Public and/or community engagement in research and scholarship,” the term community engagement is introduced, whereas previously, the term was community. This may confuse the reader who understands community and community engagement to be two different concepts. There is also a need to make a clear distinction between public good (as in public commodity) and public good (as in common good, altruistic mission etc.).

2) A good number of the institutions that are included in the study are institutions that have received the Carnegie Foundation's Community Engagement Elective Classification. In several of the quotes, the institutions were using definitions that were almost exactly the same as Carnegie's. Please make this clear in the text: this is relevant context because part of the application process for the elective classification asks about RPT and definitions of community engagement. Some institutions have used the Carnegie classification process as a way of informing and influencing revisions to RPT guidelines.

Methods and analysis

3) Please explain the rationale behind the choice to use a 15-word limit for the word frequency analysis.

4) It would be helpful for readers to have more detail about how NVivo was used to count mentions; at the very least, a description of the NVivo settings is necessary. If this information is too detailed for inclusion in the manuscript, it could be added to the documentation provided with the data.

5) With regard to the statement that the coding was refined using “all the associated variants identified by Barreno et al., 2013,” please describe these variants further (either here or at the earlier mention in the Introduction).

6) Public-status of the university should be used as a background variable, in addition to as a dependent variable. This could be done in two ways. First, universities could be classified in to two (or more) categories based on the public ownership. Second, universities could be classified as public and private (or several different categories) by the source of funding (public, private).

7) When analyzing the differences by disciplines the US and Canadian universities are considered to be part of the same population – please explain the rationale for this decision.

Connections with existing literature

8) Advocates of community-engaged scholarship have worked hard to clearly differentiate community-engaged scholarship from community service and public outreach – especially in RPT guidelines. Please include some reference to the body of scholarship on this topic. For example, over the past 20 years, we have seen various statements, “toolkits,” and standards for documenting and evaluating community-engaged scholarship in faculty RPT guidelines, including the Carnegie Foundation's Elective Community Engagement Classification application framework, the Association of Public Land-Grant Universities' Task Force on the New Engagement, Research University Engaged Scholarship Toolkit developed by The Research University Civic Engagement Network (TRUCEN), the Community-Engaged Scholarship Toolkit developed by Community-Campus Partnerships for Health, the Scholarship in Public: Knowledge Creation and Tenure Policy in the Engaged University developed by Imagining America, and a partnership of eight Canadian universities that developed Rewarding Community-Engaged Scholarship: Transforming University Policies and Practices.

Presentation of results

9) The unusual presentation of the chi-square results makes it difficult for the reader to locate important statistical information such as the degrees of freedom and the significance level for each result. I would recommend that the authors report the analytical results in a way that is consistent with standard practice (e.g., [https://depts.washington.edu/psych/files/writing_center/stats.pdf]). Please explicitly state that a chi-square was used to test independence for the results shown in Figures 1-3 and 6-8, along with the null and alternate hypotheses for each test. In each case, when stating that the conditions were not met for the chi-square test, please indicate what conditions were not met.

10) Please confirm the levels of significance shown in Figures 1 and 3: these are given 0.05, 0.01 and 0.001 in the subsection “Identification of terms,” but in the figures themselves are labelled 0.1, 0.05, 0.001. If 0.1 is correct, please explain why this has been included, as it is generally considered below the threshold for significance.

Discussion

11) The Discussion should be revised to alter the normative tone of the discussion and recommendation. Publicness is not a synonym for good and it is not especially a synonym for OA. Also traditional metrics can be considered to be a proxy of publicness of the university if we define the role of public universities as “seeking the objective truth” or as a following of “Mertonian norms of science” or as “Humboldtian universities.” The only way of providing normative policy advice for universities would be a systematic analysis of the strategies of the universities and the covariance of the RPT-documents with the strategy. Currently, the lack of this connection is more hypothetical than based on empirical findings.

Essential revisions:

Definition of terms

1) Please revise the text to be clear and consistent in your use of terms such as community, community engagement, public scholarship, and public engagements, and make sure these are all introduced in the Introduction section. This is important, particularly around the term “community engagement.” For example, in the subsection “Public and/or community engagement in research and scholarship,” the term community engagement is introduced, whereas previously, the term was community. This may confuse the reader who understands community and community engagement to be two different concepts. There is also a need to make a clear distinction between public good (as in public commodity) and public good (as in common good, altruistic mission etc.).

We addressed this point through a few changes:

First, we addressed it by being more explicit in the Introduction section when discussing community engaged scholarship (CES) and related practices. As we present in the Introduction, we see CES as one of the ways in which universities may try to address the public dimensions of faculty work (in particular, the “public good” dimension) and so bolstered the paragraph stating as much by acknowledging the growing body of work about CES and related practices, as well as referring to the statements and toolkits that the reviewers recommended.

More significantly, we have now been much more explicit about our choice of the term “public and/or community engagement” and how it is distinct from the terms “public” and “community.” In particular, we added all of the terms and further explained our strategy for coding the concept of “public and/or community engagement.” These additions can be seen at the beginning of the subsection “Public and/or community engagement in research and scholarship”. Reviewers will also notice a correction of the term in other places where we had relied on “community engagement” as a shorthand.

2) A good number of the institutions that are included in the study are institutions that have received the Carnegie Foundation's Community Engagement Elective Classification. In several of the quotes, the institutions were using definitions that were almost exactly the same as Carnegie's. Please make this clear in the text: this is relevant context because part of the application process for the elective classification asks about RPT and definitions of community engagement. Some institutions have used the Carnegie classification process as a way of informing and influencing revisions to RPT guidelines.

We agree that it is important to acknowledge the Carnegie Community Engagement classification, even if it is elective and therefore not available for all our units. We have now included it in two ways: First, by acknowledging its existence and influence in the introduction in the paragraph regarding community engagement. Second, by explicitly pointing out that some of the quotes draw very directly from this definition of community engagement. Lastly, we calculated the number of institutions that had opted into the classification and the number of them that received the designation, and included these figures in the text.

The relevant change can be seen in the subsection “Public and/or community engagement in research and scholarship” where we added: “This particular instance, and others like it, draw their definition of community engaged scholarship from the Carnegie classification described earlier, which explicitly asks if community engaged scholarship is rewarded in faculty promotion. In our sample, 85 institutions (all from the United States) had opted to have their community engagement assessed, and 34 had attained the classification.”

Methods and analysis

3) Please explain the rationale behind the choice to use a 15-word limit for the word frequency analysis.

We are not aware of a specific standard distance used in linguistics research for such analysis, so we had opted for a word limit that approximated one sentence on either side of the term of interest. The average sentence length in English is estimated at 14.4 words (The Acropolitan, 2017). We have made this rationale explicit and included a citation in the first mention of searching “near” a term of interest (subsection “Context surrounding public and community”).

4) It would be helpful for readers to have more detail about how NVivo was used to count mentions; at the very least, a description of the NVivo settings is necessary. If this information is too detailed for inclusion in the manuscript, it could be added to the documentation provided with the data.

We included a line indicating that we used the “matrix query” option in NVivo to generate the incidence matrix (subsection “Identification of terms”, second paragraph) and specified that we used code written in Python to calculate numbers. We have now also published this code and cited it at the end of that same paragraph.

5) With regard to the statement that the coding was refined using “all the associated variants identified by Barreno et al., 2013,” please describe these variants further (either here or at the earlier mention in the Introduction).

This was done, as described above under point 1 in the subsection “Definitions of terms”.

6) Public-status of the university should be used as a background variable, in addition to as a dependent variable. This could be done in two ways. First, universities could be classified in to two (or more) categories based on the public ownership. Second, universities could be classified as public and private (or several different categories) by the source of funding (public, private).

We appreciate the reviewers' comments, but are not sure this distinction is significant in this case. In the Introduction, we mention how both public and private universities have public dimensions (e.g., private institutions receive public funds for research, and public institutions receive corporate donations), and agree with the analysis of Simon Marginson, one of the most influential Higher Education scholars, who argues that defining public/private universities in terms of legal ownership does not advance our comprehension in terms of the special characteristics of higher education as producers of knowledge. Instead Marginson suggests that we should focus on the social character of the goods produced by an organization when he argues “that public/private goods are not always zero sum and under certain conditions provide conditions of possibility for each other”. Perhaps a more important distinction here would be whether the institutions are profit or nonprofit, but in our sample, there are no for-profit colleges or universities to analyze.

Marginson, S. (2011). Higher education and public good. Higher Education Quarterly, 65(4), 411-433.

7) When analyzing the differences by disciplines the US and Canadian universities are considered to be part of the same population – please explain the rationale for this decision.

Given the similarity between the RPT processes in both countries, our preference throughout the analysis was to consider all the institutions as part of a single population. Such a treatment is commonly found in the literature (for a few examples see Goldstein and Bearman, 2011; Genshaft et al., 2016; Gruzd, Staves and Wilk, 2011). Where we have deviated from this practice is in the analysis of institution subtypes (i.e. R1, R2, R3 and RCan). The decision to keep the Canadian institutions separate was borne from the lack of commonly accepted subtype classification of the Canadian institutions. That is, we had no basis with which to classify Canadian R-types as being either R1, R2, or R3.

Genshaft, J., Wickert, J., Gray-Little, B., Hanson, K., Marchase, R., Schiffer, P., & Tanner, R. M. (2016). Consideration of Technology Transfer in Tenure and Promotion. Technology and Innovation, 197–20. https://doi.org/10.3727/194982416X14520374943103

Connections with existing literature

8) Advocates of community-engaged scholarship have worked hard to clearly differentiate community-engaged scholarship from community service and public outreach – especially in RPT guidelines. Please include some reference to the body of scholarship on this topic. For example, over the past 20 years, we have seen various statements, “toolkits,” and standards for documenting and evaluating community-engaged scholarship in faculty RPT guidelines, including the Carnegie Foundation's Elective Community Engagement Classification application framework, the Association of Public Land-Grant Universities' Task Force on the New Engagement, Research University Engaged Scholarship Toolkit developed by The Research University Civic Engagement Network (TRUCEN), the Community-Engaged Scholarship Toolkit developed by Community-Campus Partnerships for Health, the Scholarship in Public: Knowledge Creation and Tenure Policy in the Engaged University developed by Imagining America, and a partnership of eight Canadian universities that developed Rewarding Community-Engaged Scholarship: Transforming University Policies and Practices.

We have brought in mentions of these statements and toolkits, especially the Carnegie Foundation’s Elective Community Classification, along with reference to a full bibliography on community engagement in addressing the earlier point regarding the definition and use of the terms “public,” “community,” and “public and/or community engagement.” While our work does not delve into or contribute to the fine distinctions made in the literature, we hope that our more careful treatment and use of the terms, along with the clearer links to the established definitions, address the underlying concern of the reviewers.

Presentation of results

9) The unusual presentation of the chi-square results makes it difficult for the reader to locate important statistical information such as the degrees of freedom and the significance level for each result. I would recommend that the authors report the analytical results in a way that is consistent with standard practice (e.g., [https://depts.washington.edu/psych/files/writing_center/stats.pdf]). Please explicitly state that a chi-square was used to test independence for the results shown in Figures 1-3 and 6-8, along with the null and alternate hypotheses for each test. In each case, when stating that the conditions were not met for the chi-square test, please indicate what conditions were not met.

We have now added more careful reporting of the Chi-square analysis following the reviewers’ suggested guidelines. The additions can be seen at the end of the paragraphs following Figures 1, 3, 6, and 8.

10) Please confirm the levels of significance shown in Figures 1 and 3: these are given 0.05, 0.01 and 0.001 in the subsection “Identification of terms,” but in the figures themselves are labelled 0.1, 0.05, 0.001. If 0.1 is correct, please explain why this has been included, as it is generally considered below the threshold for significance.

Thank you for identifying this typo. The value used was 0.01, but was incorrectly written as 0.1 in the figure legend. These have been corrected.

Discussion

11) The Discussion should be revised to alter the normative tone of the discussion and recommendation. Publicness is not a synonym for good and it is not especially a synonym for OA. Also traditional metrics can be considered to be a proxy of publicness of the university if we define the role of public universities as “seeking the objective truth” or as a following of “Mertonian norms of science” or as “Humboldtian universities.” The only way of providing normative policy advice for universities would be a systematic analysis of the strategies of the universities and the covariance of the RPT-documents with the strategy. Currently, the lack of this connection is more hypothetical than based on empirical findings.

Thank you for this suggestion. While we thought we had been careful to avoid such language, on rereading the Discussion we agreed with the reviewers about the normative tone. We have revised our Discussion section to remove instances of such language and to be more careful about how we frame our interpretations and our suggestions for what might be done.

Author details

1. Juan P Alperin

   Juan P Alperin is in the School of Publishing and the Scholarly Communications Lab, Simon Fraser University, Vancouver, Canada

   Contribution

   Conceptualization, Resources, Formal analysis, Supervision, Funding acquisition, Visualization, Methodology, Writing—original draft, Project administration, Writing—review and editing

   For correspondence

   juan@alperin.ca

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-9344-7439

2. Carol Muñoz Nieves

   Carol Muñoz Nieves is in the Scholarly Communications Lab, Simon Fraser University, Vancouver, Canada

   Contribution

   Data curation, Formal analysis, Writing—review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-8857-3000

3. Lesley A Schimanski

   Lesley A Schimanski is in the Scholarly Communications Lab, Simon Fraser University, Vancouver, Canada

   Contribution

   Conceptualization, Formal analysis, Supervision, Methodology, Writing—original draft, Project administration, Writing—review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-4664-179X

4. Gustavo E Fischman

   Gustavo E Fischman is at the Mary Lou Fulton Teachers College, Arizona State University, Tempe, United States

   Contribution

   Writing—original draft, Writing—review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-3853-9856

5. Meredith T Niles

   Meredith T Niles is in the Department of Nutrition and Food Sciences, University of Vermont, Burlington, United States

   Contribution

   Conceptualization, Funding acquisition, Methodology, Writing—review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-8323-1351

6. Erin C McKiernan

   Erin C McKiernan is in the Departamento de Física, Universidad Nacional Autónoma de México, Mexico City, Mexico

   Contribution

   Conceptualization, Supervision, Funding acquisition, Methodology, Writing—review and editing

   Competing interests

   is a member of the DORA Steering Committee and an advisor for the Metrics Toolkit, both volunteer positions.

   "This ORCID iD identifies the author of this article:" 0000-0002-9430-5221

• 5,331

  views

• 518

  downloads

• 97

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.