At eLife, we strongly support the use of preprints for sharing research results. We encourage early dissemination and scholarly discussion of interim research products, which allow investigators to truly accelerate discovery. We have shared our views with the NIH in their recent call for feedback on the subject. Here we enclose our full response to the consultation.
--------------------------
1. Types of interim research products your or your organization create/and or host.
eLife is an open-access biomedical research journal, supported by three international funding agencies (Howard Hughes Medical Institute, Wellcome Trust, Max Planck Society).
Improving the communication of scientific findings is at the heart of eLife’s mission, and we believe that results should be made available as quickly and as widely as possible. We encourage authors who submit their work to eLife, to deposit manuscripts as preprints before peer review in order to increase access to research findings and to communicate new results more quickly. There is also a mechanism to transfer manuscripts from bioRxiv directly to eLife.
We remain convinced that thoughtful peer review adds considerable value to individual manuscripts and to the scientific literature as a whole. We have now published several articles in eLife that were previously shared via preprint servers such as arXiv or bioRxiv.
eLife is also publishing the results of the Reproducibility Project: Cancer Biology- a collaboration between the Center for Open Science and Science Exchange to independently replicate selected results from a substantial number of high-profile papers in the field of cancer biology. For each paper a Registered Report detailing the proposed experimental designs and protocols for the replications is peer reviewed and published prior to data collection. The results of these experiments will be published in a Replication Study. The project will provide evidence about reproducibility in cancer biology, and an opportunity to identify factors that influence reproducibility more generally. We have currently published 29 Registered Reports as part of this initiative and will shortly be publishing the first of the Replication Studies themselves.
2. Feedback on what are considered to be interim research products, and how they are used in your field.
The key feature of a preprint that makes it an ‘interim’ product is that it has not been peer reviewed. Many preprints go on to be published - after undergoing peer review and revision - in a journal.
Having posted a preprint, the authors might also receive constructive suggestions for improvements to the article which can then be addressed in a revised version of the preprint. This process can happen in parallel with formal peer review, which means that the final published version can address comments made on the preprint and comments made by journal editors and referees.
A Registered Report is an interim research product in the sense that it details proposed experimental designs and protocols before data is collected. However, it differs from a preprint in that it is peer reviewed (although, of course, the initial version of a Registered Report can be shared as a preprint before peer review). In the RPCB the results of these experiments will be published as a separately peer-reviewed Replication Study. The general idea behind the Registered Report format is that the results will be published regardless of the direction or strength of the findings.
The key point then is that an interim product is an early version of a piece of work that is likely to be revised and published after peer review.
We strongly support the view that applicants should be encouraged to cite all relevant research outputs in their grant and job applications and research reports, including interim products such as preprints, as well as datasets, code and other resources.
3. Insight on how particular types of interim research products might impact the advancement of science.
The deposition of manuscripts in preprint repositories before journal submission offers several benefits to authors and readers. Authors can release their findings early and make them openly available to any interested reader, and so gain credit for their achievements without having to wait for the journal publishing process, which takes months or even years, to complete. This is particularly beneficial for early-career researchers seeking to demonstrate their productivity to funders and future employers in a timely manner. They can also receive suggestions and feedback from the broader community and revise their reports accordingly. Similarly, interested readers will see relevant work earlier, which could help their own research programs. Another potential benefit is that the widespread circulation of preprints will shed light on the value that formal peer review provides and could stimulate additional experimentation with peer review processes.
The support and encouragement for preprint deposition is in line with the recent call for publishers to make vital research rapidly and openly available during a public health emergency. As a signatory on the statement published by the Wellcome Trust on February 10, 2016, eLife supports the mechanism by which preprint publication allows the research community to communicate important findings as soon as possible. We see no reason why these practices should not be applied to science as a whole and hence encourage all authors to adopt this approach.
Beyond preprints, there are substantial benefits to broadening the research outputs that can be considered as first-class research objects in research evaluation. Such an approach will help to incentivise the more rapid, thorough and effective sharing of research findings and resources.
4. Feedback on potential citation standards.
The proposed standards for interim prodocts suggested in the RFI seem very appropriate:
- “Ensuring the document is preserved, findable, and freely accessible to people and machines
- Links to other versions and associated data and resources
- Attribution and disclosure of authorship, funding, competing interests, licensing, and other issues used in high-quality scholarly publication
- A clear statement that the product is preliminary, and the level of peer-review it has received (if any)”
It seems reasonable to apply the same criteria to the formally published articles that are included in an application (notably that the work be freely accessible) and to any other research products that are cited (such as data and other resources).
It is noteworthy that Crossref have recently announced mechanisms to ensure that preprints are properly supported as fully citable parts of the scholarly record.
Additional requirements. For research involving humans and animals, the preprint should indicate that the work has undergone appropriate ethical review and that the necessary approvals have been obtained. Researchers could provide a stable and persistent identifier to the research product (eg DOI). It might also be worth asking researchers to ensure that these products are all included in their ORCID profile. Finally, researchers could submit evidence about the usage of the research products, using qualitative and quantitative indicators such as the impacts on other studies, citations, published commentary about the work, media mentions, influence on policy or practice, and so on.
5. Insight on the possible need and potential impact of citing interim products on peer review of NIH applications.
There is a widespread perception that current research evaluation practices focus far too narrowly on publications in high-profile peer-reviewed journals. The perverse incentives that arise from this approach have been highlighted by researchers, policymakers and funders, and there have been calls for broadening the approach to research assessment (eg Metric Tide Report, San Francisco Declaration on Research Assessment, Rescuing US biomedical science from its systematic flaws) to include a range of outputs and accomplishments.
Broadening the range of research products, such as preprints, that can be included in applications and reports will provide a richer picture of the accomplishments and contributions of researchers. Such approaches will help to incentivise greater transparency and sharing in science and could also help to support the integrity and reliability of the published literature. The peer review process will be enhanced because of the greater diversity of information that will be available to assess researchers and the work that they are proposing.
6. Advice on how NIH reviewers might evaluate citations of interim research products in applications.
It might be helpful to limit the number of cited items that scientists include in their applications and reports, so that they focus on the outputs that have been or are likely to be the most promising. Evaluation would then be assisted by asking scientists to include qualitative and quantitative metrics about any of the outputs that they are citing, as suggested above. Similarly each output cited could be accompanied by a short ‘impact statement’ which summarises the value of the work. Such a body of evidence could assist the NIH reviewers in their in-depth evaluation of an application or a report. Preprints should be evaluated in the same way as a peer-reviewed article. Effective evaluation will require a deep understanding of the work, expert judgment and sufficient time to come to such a judgment. The reviewers’ evaluations should themselves also be recognised as critical contributions to the scientific process.
7. Any other relevant information
The sharing of preprints is a well-established and highly-valued practice in some fields such as physics and mathematics. In the biomedical sciences many of the benefits of preprint sharing will be similar. However, the direct relevance to people’s lives of some areas of medical research in particular means that it is very important to label work appropriately as not peer-reviewed.
In the longer term, and as the corpus of preprint content grows, it will be possible to provide additional services, such as text mining, to explore the preprint literature. Another possibility is to explore the addition of ‘overlay’ peer-reviewed journals onto the preprint literature, which is currently being tested for example in the field of mathematics using the well-established arxiv.org repository. These and other innovative approaches could have a dramatic and positive impact on the efficiency and cost-effectiveness of research communication.
We strongly support moves by the NIH to encourage researchers to include preprints and other research outputs in their funding applications and reports.
