Responsible Research Assessment of Faculty: maximizing quality, transparency, and trustworthiness of scientific research in Research-Performing Organizations

Reviewer #1 (Public review):

Summary:

This study addresses the gap between calls to incentivize Open Science practices through research assessment reform and its implementation by Research Performing Organizations (RPOs). To help RPOs get started and prioritize their assessment reforms, it makes a series of recommendations for action. It does so via an expert Delphi process, where consensus around six standards was reached. The purpose is to provide a resource for would-be reformers in Research Performing Organizations around the world, in the hope of advancing the Responsible Research Assessment and Open Science reform movements across global academia. The standards are not meant to be one-size-fits-all, but customizable to local needs and implemented selectively at the discretion of a given RPO.

Strengths:

Overall I was impressed with the contribution. Generally speaking, it clearly sets out its methods and arguments. I recognize there is indeed a growing need to support RPOs at varying levels of maturity, to get started and orient themselves around research assessment reform, so this seems to me like it will be a useful offering. The Delphi approach is well explained, and further information is provided in supplementary files.

Weaknesses:

Some further reflection on the disciplinary make-up of the Delphi participants would be helpful. Somewhat more hedging on the limitations of the manuscript would help improve the contribution, as would further details on how this contribution sits in relation to existing resources and studies on implementing open science-aware research assessment reforms.

Reviewer #2 (Public review):

Summary:

This paper describes the employment of a Delphi process, a common consensus-reaching practice based on consecutive rounds of discussion and voting. In this study, the authors engaged in the Delphi process with stakeholders to reach consensus on recommended guidelines for use in tenure and promotion, specific to reproducible and transparent research practices.

Strengths:

The paper provides very practical guidelines for research institutions willing to push for reform regarding reproducible and transparent practices. Guidelines like these are important for institutional leadership that wishes to drive change at their institutions but does not necessarily hold topic expertise on research and reproducibility. The study's authors represent longstanding expertise on the topic, and the use of the Delphi process ensures that stakeholders were engaged in the recommendations, something that helps ensure uptake and buy-in.

Weaknesses:

The biggest weakness of the paper is a lack of engagement with the literature of tenure and promotion reform. This is a field that has been heavily studied with regard to incorporating change in practice to promote equity and inclusion. Other specific subfields that have dealt in this space are those that have pushed for reform to increase the importance of teaching, service, and (inclusion of) mentorship in tenure and promotion packages. Because this paper deals heavily with implementing change in this specific space, and provides guidelines with very practical implications for change, some literature review in the introduction and discussion on where past efforts have succeeded or faced roadblocks would be welcome. This would add to the study's value in helping people implement these practices in a very real and tangible way. The other weakness is a lack of further expansion on the recommendations that did not reach consensus, many of which were added in Round 3. Are more rounds warranted for future studies? Changing landscapes require changing recommendations, which is probably why things like the use of LLMs emerged later and did not reach consensus. How should the readers engage with these results, and what future steps are needed to implement guidelines (if any) around these topics?

Reviewer #3 (Public review):

Summary:

This work provides a novel guide for Research Performing Organizations (RPOs) to evaluate researchers in ways that promote responsible research practices. Such a tool could help inform the development of policies that foster these practices within RPOs and potentially influence decisions related to the hiring and promotion of scientific staff. The guide was developed through a modified three-round Delphi process based on a preliminary set of recommendations provided by members of various organizations. Each round included a survey, and items that did not reach a consensus of >80% were reconsidered in the subsequent round. Round 3 consisted of an in-person meeting, during which the final guide was developed based on criteria that reached {greater than or equal to}80% consensus among participants. The guide includes six practices for evaluation (verification efforts, prospective study registration, data/code/materials sharing, reporting transparency, open-access publishing, and disclosure of funding interests) with an additional seventh component. Each practice includes a clearly defined aim and criteria describing what good practice looks like. The complete study methodology, statistical analyses, and supporting data are available through Open Science Framework (OSF).

Strengths:

A major strength of this work is the development of a practical tool that could help foster responsible research practices among candidates and provide promotion and hiring panel members with clearer criteria for assessing such practices. The three-round Delphi process, which included international participants, provides a structured approach to reaching consensus on practices that could contribute positively to the research ecosystem.

Weaknesses:

The recruitment strategy may have introduced selection bias and limited the representation of researchers and other interest holders from diverse geographical and institutional contexts, particularly from the Global South. Although the authors acknowledge that variability across regions and institutions should be considered when applying the guide, greater representation of these contexts would strengthen confidence in the broad applicability and generalizability of the proposed framework.