By Mark Patterson, eLife Executive Director
Today, eLife is launching a trial to test the feasibility of a radical form of peer review. The essential idea is that, once an editor has invited a manuscript for full peer review, the journal is committed to publishing the work along with the reviewer reports, the decision letter, and the author response. The trial is based on ideas that were first presented by Erin O’Shea and colleagues from the Howard Hughes Medical Institute at the ASAPbio meeting earlier this year. The background and motivations for the trial are discussed in an accompanying Editorial by Mark Patterson and Editor-in-Chief Randy Schekman. The essence of the trial is as follows:
- New submissions are initially evaluated by Senior Editors as normal and selected articles are invited for peer review.
- After peer review, and consultation amongst the referees, the Editor compiles a consolidated decision letter for the author.
- The authors decide how to respond to the decision letter and submit a revised manuscript and a response to the decision letter and reports.
- The Editor evaluates the response and the revised article is published along with the decision letter, the full review reports, the author response and an assessment by the Editor (see below).
How the trial will work is described in detail below. To be kept up-to-date on our progress, sign up to be notified by email.
Taking the decision
Once the editorial leadership of eLife (the Editor-in-Chief and Deputy Editors) had decided to consider conducting a trial along the lines proposed, the next step was to consult the Senior and Reviewing Editors of eLife. We received responses from many editors (around 80 in total), and a clear majority were in favour of conducting the trial.
Editors were attracted by various aspects of the trial, including the idea of placing greater responsibility and accountability in the hands of researchers as authors, and the potential for referees to gain credit for thoughtful criticisms or creative suggestions. The most common theme in the supportive comments was the idea of doing an interesting experiment in peer review, and one that might have a positive impact on scientific culture.
The most frequent concern expressed by editors was about the impacts on the initial decision step in which the Senior Editors decide whether articles should be invited for full peer review. In the trial, the decision to proceed with peer review is essentially a decision to publish the article, and so the concern is that the initial decision stage might become more selective with more manuscripts being rejected. The current rejection rate at the initial decision step is around 70%.
The second concern was about publishing articles where the peer review process judges that the work is technically flawed or less valuable to the field than judged. Currently around 45% of articles are rejected after peer review. However, given that the peer review reports, decision letter, and author response are all published along with the article, it is in the authors’ interests to respond fully and constructively to all comments. We also know that the great majority of articles rejected after peer review at eLife are later published in other peer-reviewed journals.
These and other issues will be explored as we conduct the peer review trial over the coming months and we will share the results – along with other metrics and indicators we’ll be monitoring, which are described below.
Process of the trial in detail
Initial submission
- Authors of all new submissions are given the option to participate in the trial by submitting their work as a specific article type (Research Communication).
- The trial is limited to the first 300 such articles.
- Deputy Editors and Senior Editors are assigned and informed if the article is part of the trial.
- The Senior Editor assesses the article (with Reviewing Editor input as appropriate) and decides whether or not to review the article, using the same criteria as now.
- If invited for full review, we are committing to publishing the article.
Full submission
- Upon full submission, the article goes through our usual quality control processes. All articles must meet eLife policies for research and publication ethics, data availability, etc.
- Reviewing Editor recruits referees who are informed of the trial process.
- Instructions to referees indicate that the full reports will be published with the article.
- Once reports have arrived the consultation amongst referees is started as usual.
- As happens at present, extra experiments/work should only be requested if they are essential to support the claims being made in the paper.
- However, unlike at present, it will be acceptable to recommend experiments that will take longer than two months to do.
- The decision letter summarises the following:
-
- Main strengths of the work
- Major concerns that require revision in the manuscript and/or a response from the authors
- Any other suggestions that the editor wishes to highlight
- The decision letter and the full reports are sent to the author.
Revision
- The authors submit a revised manuscript and their response to the decision letter. (Journal staff will send the revision submission back to authors if the major concerns in the decision letter are not addressed in the revised manuscript or the author response.)
- The editor, in consultation with the reviewers if necessary, assesses how the authors have responded to the issues raised during peer review and decides between the following three options: all the issues have been addressed; minor issues remain unresolved; or major issues remain unresolved. This assessment is conveyed to the authors in the acceptance letter.
- The editor’s assessment will also be published as an “Editorial note” in a prominent position at the end of the abstract in the paper.
- The article could go through another round of revision, for example if the editor/referees feel that there has been a misunderstanding or there is an obvious issue that could be fixed.
Publication
- The revised article, decision letter (with full reports), author response, and the editor’s assessment are then published.
Example of an article abstract showing the editor’s assessment of the author response to peer review.
Evaluating the trial
To help measure the effectiveness of the trial and support for this type of approach, we’ll collect the following data.
Interest in and response to trial process
- % authors who opt in to the trial
- % of referees who accept invitation to review
- % revised articles that address all the concerns
- Views on the trial process from editors, referees, and authors (via interviews and surveys)
- External commentary
Other editorial impacts
- % articles invited for full peer review (current rate is ~30%)
- The overall acceptance rate (currently ~16%)
- % of appeals for papers that are rejected after initial submission
- % referees who sign reviews
- Median time to acceptance
- Web metrics about usage of the articles, including how frequently readers consult the decision letter
Our main goals are to discover the strengths and weaknesses of the approach (as well as any unintended consequences), and to share these findings.
Questions and comments are welcome. Please annotate publicly on the article or contact us at hello [at] elifesciences [dot] org.
For the latest updates on the trial and other news from eLife, sign up to receive our bi-monthly newsletter. You can also follow @eLife on Twitter.
