Jody Culham is a professor of psychology at Western University in Canada and an eLife Reviewing Editor who has been with the journal since it was launched in 2012. Before then she had started to feel that the scientific publishing system contained fundamental flaws which meant that it was no longer working in the best interest of researchers, and instead seemed to set editors and reviewers against authors. Culham sees the role of an editor as "helping those papers that will move their field forward to shine as they deserve". Now, after handling around 40 submissions as a Reviewing Editor and personally reviewing a total of 18 manuscripts for eLife, she feels that "a consensus approach enhances the merits and limits the pitfalls of peer review".

The ultimate goal of the reviewer consultation is to agree upon a clear list of revisions that the authors would need to complete in order to get their paper accepted. Davis Ng, an associate professor at the National University of Singapore, was impressed with the online consultation when he first reviewed a paper for eLife in 2014. After reviewing a second manuscript, he was invited to become a Reviewing Editor for papers in biochemistry and cell biology in July 2015. Ng feels that the reviewer consultation helps him perform his duties as an editor. "Consulting with the reviewers makes so much more sense to me than trying to resolve differing opinions from static written reviews," he explains.

Stephen Royle – an associate professor at Warwick Medical School – has published two papers in eLife and described the review process as "hassle-free". For his second paper, Royle recalls how he appreciated the way in which the decision letter clearly explained how the manuscript needed to be revised, and how he was able to submit the revised manuscript 54 days later. However, unbeknown to Royle at the time, the five essential revisions listed in the decision letter were actually the end result of 22 comments going back and forth between the Reviewing Editor and the three reviewers over the course of several days. "I didn't realize until later that there had been some disagreements among the referees during the discussion," says Royle. "The collaborative process really helped to iron these out."

Most reviewers (95% in fact according to a 2016 survey) agree that the consultation process at eLife adds value for authors. "It gets the authors actual intelligible and unified feedback on their work," says Vosshall, who has reviewed nine manuscripts for eLife and published one paper in the journal. Alexey Merz, an associate professor in biochemistry at the University of Washington, has also experienced the process from both sides and thinks it is the best peer-review process of those he has encountered: "Not all decision letters will achieve their intended goal equally well, but it is great to see a process designed to encourage a constructive, actionable report rather than a laundry list."

So, how easy is it for reviewers to reach an agreement? Maddy Parsons is a professor of cell biology and director of Nikon Imaging Centre at King's College London and was one of 70 new Reviewing Editors welcomed to eLife in 2016. She says that, so far, she has been "pleasantly surprised by how open most reviewers are to compromising on some issues they raised when the general consensus in the consultation goes the other way."

Culham has also been able to reach a consensus that was acceptable to all the reviewers for every manuscript that she has handled. She recalls a situation in which one reviewer said, "[this result] makes no sense; take it out", while another reviewer said, "[this result] makes no sense; discuss it more". Instead of relaying these contradictory statements to the authors, as would probably happen at most journals, Culham and the two reviewers conferred, agreed that expanding the discussion of the "surprising" results was best, and suggested how the authors could approach doing this.

The consultation sessions also give reviewers a chance to have deep, and sometimes intense, discussions about papers with colleagues at other institutions. "I'm generally unsatisfied by the 'what did you think of this recent paper?' type of conversation at conferences," says Megan Carey, a group leader at Champalimaud Neuroscience Programme in Lisbon. "When I discuss papers with people I like to be fully immersed in them." Carey first reviewed a manuscript for eLife in 2014: "I loved it," she recalls, "it felt almost like a journal club, something you normally only do with colleagues at your home institution, who might be in different fields."

Carey has since reviewed four more papers for eLife and says her experience as a reviewer influenced her decision to submit a paper to the journal in 2015. She describes the paper as "somewhat unusual" in that it cut across multiple subfields of neuroscience. She chose to submit to eLife because she hoped that the consultative process might allow reviewers from different backgrounds to appreciate the contributions the paper had made, both methodologically and scientifically. Though the paper’s path through peer review was far from straightforward (it was initially rejected after discussions between the reviewers and later accepted after an appeal), Carey feels that she got a better result in the end than if she had submitted to another journal. Also, because the decision letter and author response from the original rejection and eventual acceptance are published alongside the paper, Carey likes the fact that "readers can actually decide for themselves about the fairness of the process".

Though authors generally appreciate a clear decision, some have suggested that having the reviewers reach a consensus might not always be the best way to improve the manuscript. Reviewers will interpret the same paper in different ways and some authors appreciate seeing any feedback that could help them to strengthen their manuscript. As Merz puts it: "divergence among referees can be useful too."

The benefits of the consultation do, of course, come with a cost. "Because each step in the process involves several scientists, there can be delays," says Ng. To compensate for this extra time, eLife editors aim to rapidly triage initial submissions to decide which papers should move forward to full peer review. Culham points out that, while this process is not always perfect, it helps to preserve the time of editors and reviewers for those manuscripts that show the most promise. "It also," she adds, "reduces the likelihood that authors waste time waiting for a negative outcome."

Between them, Culham, Parsons and Ng have been consulted on over 400 initial submissions and always aim to put aside blocks of time to read new manuscripts and share their thoughts on them as soon as possible. Parsons often looks at initial submissions on the train during her 40-minute commute to and from work. "You need to respond quickly to requests when they drop into your inbox," she says, "which includes saying if you feel you are not qualified to judge the science. Senior Editors and authors really appreciate fast, honest responses."

While the consultation about a manuscript inevitably delays the decision by several days or more, Parsons believes that spending time to clarify exactly what is needed in the revision is an efficient use of time. "It actually speeds up the process later on, by preventing endless rounds of miscommunication and re-reviewing," she says. Indeed, over 70% of the papers published in eLife were accepted after a single round of peer review.

Editors can also take steps to minimize the delay caused by the consultation itself. After reading all the reviews, Parsons tries to structure the online consultation to make the most effective use of the reviewers' time. She assembles a list of the core points from all the reviews, and identifies potentially unnecessary or unreasonable requests that should probably be excluded from the decision letter. She then uses these summaries to begin her discussion with the reviewers. Parsons says that, in her experience, this proactive approach "helps to steer the consultation process in the direction of a timely and well-rounded response". She adds that the process is also "smoother for everyone" when all the reviewers are communicative and prompt to engage in the discussion.

Beyond the extra time required for the eLife peer review process, some have expressed concern about reviewers being identified to one another during the consultation (Schekman et al., 2013). For example, some question if a junior investigator might feel reluctant to challenge the views of an established expert, and suggest that anonymity would allow reviewers to more freely say what they think and also reduce the influence of biases. However, most eLife editors think that transparency is an essential part of the process. Culham, for example, finds that it is easier to reach a consensus "when you know who you are interacting with". She also believes that when the reviewers expect that they might have to justify their reviews to their peers, they "conduct thoughtful reviews and limit idiosyncratic requests". Parsons adds: "I think it also helps reviewers curtail the temptation to ask for endless, unnecessary experiments that will only delay publication."

Culham disagrees with the suggestion that consultative peer review disadvantages early-career reviewers. "Generally, I've found reviewers very respectful of one another regardless of career stage," she says. "Like the other reviewers, junior investigators can give their honest opinion in their reviews before seeing other reviewers' reports or knowing their identities." Culham also watches out for salient points from any reviewer that might get "ignored or drowned out in the discussion" and makes sure to include them in the decision letter. Based on their responses to a survey, most reviewers agree that openness benefits the consultation process.

In its first five years eLife has published more than 3,500 manuscripts (and reviewed around twice this number), and now receives over 600 new submissions each month. This means that, either as reviewers or authors, more and more scientists are experiencing consultative peer review. Parsons feels that this exposure is helping to change the landscape of peer review.

Culham adds that her role at eLife has already changed the way she writes reviews for all journals: "I aim to be decisive and clear about necessary versus optional changes; I also admit when I'm less certain about something and would appreciate discussing it with others." Going forward, her biggest hope is that as the approach becomes more widespread, the culture of peer review will change to make it better for all involved and better for science.

Indeed, some journals – including Development, The Plant Cell and eNeuro – have already adopted a consultative review process, in one form or another, and others are experimenting with it (Bernard and Picciotto, 2016; Merchant and Eckardt, 2016; Pourquié and Brown, 2016). And after experiencing consultative peer review at eLife as a reviewer and an author, Raymond Goldstein has urged the American Physical Society to adopt the same approach for its journals (Goldstein, 2016).

In recent years, physicists introduced – or reintroduced (Cobb, 2017) – biologists to the idea of preprints as a way to make their results available as quickly and as widely as possible, while also mitigating some of the pitfalls of peer review. It would be almost poetic if the life sciences could soon return the favor by introducing consultative peer review to the physical sciences and beyond.

This Feature Article is part of a collection of articles on peer review.

1. Received: September 20, 2017
2. Accepted: September 20, 2017
3. Version of Record published: September 28, 2017

© 2017, King

This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.