By George Perry (Pennsylvania State University, eLife Reviewing Editor) and Maria Guerreiro (eLife Journal Development Editor).
eLife announced a trial in June 2018, led by Reviewing Editor George Perry, in which nine Reviewing Editors in the areas of genomics and evolutionary biology committed to involve an early-career reviewer in the peer review of each manuscript they handled over the course of a year.
In 2016, eLife had introduced an early-career reviewer pool with the aim of offering more opportunities to early-career scientists to participate in eLife's review process. However, for wide adoption and effectiveness, the pool needed to be expanded in many areas and with better ways for Reviewing Editors to identify the most relevant early-career reviewers for any particular submission.
In the trial we wanted to learn how eLife might scale up such processes, while giving early-career investigators a valuable opportunity for professional development and Reviewing Editors access to an expanded pool of invested expert reviewers.
The trial was open for self-nominations via completion of a brief form, and from 450 nominations, 290 early-career reviewers meeting the eligibility criteria were selected for the pool. Specifically, researchers had to either be a postdoctoral researcher or have spent less than five years in an independent research position, plus have a strong background in the areas of genomics or evolutionary biology we usually assess and publish in eLife. The selected individuals will remain in the eLife early-career reviewer pool beyond the completion of the trial.
In addition to George Perry, the following eLife Reviewing Editors were involved: Chris Ponting (University of Edinburgh), Molly Przeworski (Columbia University), Antonis Rokas (Vanderbilt University), Wenying Shou (Fred Hutchinson Cancer Research Center), Christian Landry (Université Laval), John Long (Flinders University), Magnus Nordborg (Gregor Mendel Institute of Molecular Plant Biology) and Kevin Verstrepen (KU Leuven).
Now that the trial has concluded, we are sharing our experiences.
Results
In the period from May 2018 to May 2019, 24 out of 29 submissions (82.7%) that were identified for inclusion in the trial included an early-career researcher in the review process. In one case where an early-career reviewer was not included, the paper was a resubmission of a previously rejected article and sent to the original reviewers for re-assessment. In another case, reviewers were shortly invited after an editor agreed to take part in the trial, and the submission had not yet been properly flagged in the editorial system as being considered for the trial. One submission ended up being excluded due to a non-returned review from the assigned early-career referee.
Overall, 27 early-career researchers performed a total of 38 peer reviews (including re-reviews). For the same manuscripts, 25 non-early-career reviewers performed a total of 42 peer reviews (including re-reviews). We note that some early-career reviewers (n=4) and non-early-career reviewers (n=5) reviewed more than one submitted manuscript in this dataset.
With the caveat that the numbers are small, we evaluated whether there were any differences between the early-career reviewer (n=27) and non-early-career reviewer (n=25) groups.
Early-career investigators were slightly quicker to return their initial review reports than their counterparts (average of 14.2 days for early-career reviewers and 16.9 days for the other reviewers). Both groups showed approximately similar levels of engagement in the consultation process that takes place after peer review in eLife.
In the interests of transparency, eLife encourages peer reviewers to disclose their names to authors. In the trial, early-career reviewers disclosed their names 23.5% of the time, and non-early-career reviewers 28.9%.
eLife Reviewing Editors can rate the reviews as Outstanding, Good, Average, or Disappointing once the peer review process is complete and a decision letter submitted. For early-career reviewers, 16 of the 28 reviews (57.1%) that were rated were deemed to be Outstanding, which was very similar to what was observed for non-early-career reviewers (9/16 = 56.2%).
Feedback from editors and early-career reviewers
Christian Landry (Université Laval), one of the eLife Reviewing Editors involved, said that: "Reviews and discussions with early-career reviewers were very rigorous, constructive and helpful in reaching decisions, and contributed to improving the manuscripts that were accepted for publication." Reviewing Editor Antonis Rokas (Vanderbilt University) added that "serving as one of the editors in the early-career reviewer trial was a transforming experience – I was so impressed with each and every one of the participants that now my first task every time I handle a manuscript is to seek out one or two early-career reviewers on the topic."
We also solicited feedback from the 27 early-career researchers who participated in the trial and received 14 responses. All respondents reported enjoying their participation in the eLife review process.
For example, Luisa Pallares (Princeton University) said: "I like the accountability that comes with the fact that your review is not only going to be read, but also discussed by the other reviewers and the editor. [...] The fact that your comments and identity are shared with the other reviewers is for me a great incentive to do a more conscientious job. [...] The second thing I really like is the possibility to discuss/agree/clarify other reviewer's comments. Sometimes, weak/strong points of a paper are misunderstandings, and this can be easily solved during the consultation session."
Colin Meiklejohn (University of Nebraska-Lincoln) added that "simply being known to other reviewers helps to improve the tone and utility of reviews."
Anna Poetsch (St. Anna Children's Cancer Research Institute) wrote that the eLife review process "forces you to read a paper really carefully that may not be entirely in the heart of your research area. What I find especially nice about reviewing for eLife," she added, "is the process of coming to a consensus. It is really educational to see what others from a slightly different research field think about the same piece of work."
Next steps
The trial has helped to reinforce the notion of substantial advantages in involving early-career investigators as peer reviewers, both for eLife and its editors and for the investigators. A greater diversity of perspectives in the review process and the ability to invite enthusiastic reviewers with known expertise are clear benefits for the journal. Meanwhile, early-career reviewers gain insights into the review process that benefits them as authors on their own work, and have the opportunity to interact with the Reviewing Editors and other scholars in the field during the consultation process.
Based on a survey of our Reviewing Editors from across all subject areas, approximately 41% are actively looking to include an early-career researcher in the review process. However, we still need to improve and scale the tools we use to identify and nominate early-career researchers as peer reviewers. Informed by the experiences of the trial and by the feedback from our Early-Career Advisory Group, we have created an updated process for the nomination and selection of early-career researchers as peer reviewers to make it more accessible and explicit. With this new process, we can directly provide each Reviewing Editor with a custom-targeted list of early-career reviewers aligned with their expertise, in addition to access to the overall pool.
Our pilot drive (via social media and an opening blog post) to build a strong pool of early-career reviewers in genomics and evolutionary biology was quite successful. Thus, in addition to the continually-open process for nominations in any field of research published by eLife, we will periodically proceed with initiatives in specific fields with support from Senior Editors and Reviewing Editors in those respective areas. Through the combination of open and targeted processes to renewably develop our early-career reviewer database and an improved toolkit for Reviewing Editors to identify suitable reviewer candidates, we will create more opportunities for early-career researchers to participate in the eLife review process.
#
We welcome comments/questions from researchers as well as other journals. Please annotate publicly on the article or contact us at hello [at] elifesciences [dot] org.
For the latest in published research sign up for our weekly email alerts. You can also follow @eLife on Twitter.
