Libero Reviewer: The making of a user-friendly submission and peer-review platform

With the development of Libero Reviewer, eLife is working towards an alternative to the monolithic proprietary systems that currently dominate the peer-review space. In particular, we hope that the application will support other journals in their adoption of an open-access publishing model. For our own authors and editors, Libero Reviewer will streamline the editorial process significantly, accelerating scientific discovery while reducing time spent on administrative tasks. The codebase is maintained entirely in the open, within a public GitHub repository, in keeping with the organisation’s commitment to open-source software.

We took an early decision that, in order to reduce risk in the project, we would release software to users as quickly as possible. This would enable us to gather feedback on user experience (UX) while extensively testing the robustness of the technology in a production environment. We were aware that releasing any portion of the software that did not support the full life cycle of a manuscript would mean integrating with the journal’s current proprietary submission system, which would inherently carry a degree of technical risk, but this was quickly outweighed by the valuable data that real user feedback would bring, allowing us to confirm or deny our assumptions and shift our approach as necessary.

With all of this in mind, the Product team determined that the first step in eLife’s editorial workflow – the submission of a manuscript to the journal – would be the ideal first milestone deliverable for public release. It would require the design and development of a number of features, along with the underlying architecture that would form the foundations of the eventual end-to-end application. Moreover, it would reach a large audience, given the ~700 initial submissions the journal receives each month, without affecting those authors with submissions already in progress.

Moving forward, I would be keen to ensure that any stories we gather are focused on the problem space itself, rather than any particular solution that a user would like replicated by the new system.

– Hannah Drury

We began the design process by gathering a number of user stories from internal stakeholders, who were tasked with identifying what they needed to be able to carry out their daily tasks. The Editorial team, in particular, were an invaluable asset in the process, given their proximity to both authors and editors as well as being eventual users of the system themselves.

While it would be clearly impossible to craft product requirements without such stories, I was conscious that many of them were potentially borne of workarounds that had been put in place to navigate legacy systems and workflows. Moving forward, I would be keen to ensure that any stories we gather are focused on the problem space itself, rather than any particular solution that a user would like replicated by the new system. Chris Huggins (our Senior UX Designer) and I are keen to map current workflows and identify where they can be potentially streamlined, since Libero Reviewer’s clean slate offers us the perfect opportunity to reevaluate our own processes.

Libero Reviewer is built using React, a relatively new JavaScript library which has made recruiting skilled developers difficult. Until recently, half of the team working to deliver the application were based in Cambridge, with the other half working out of contractor offices in London. This made good, regular communication absolutely critical, and as a team we had to get much better at it. We were not working shoulder to shoulder, so to compensate we introduced routines such as a morning stand-up and afternoon updates as ways of formally reconnecting and letting each other know what we were working on, as well as providing prime opportunities to request help if necessary. Outside of these times, we used Slack to keep in regular contact, but even this real-time method of communication requires discipline. As a Product Manager, it is important for me to understand what the developers are working on at any one time so that I can re-prioritise as necessary, and clearly this information is less easily obtained from colleagues who are not co-located.

As the project has progressed, the team itself has also grown and adapted, with members joining and leaving at various intervals. This has in turn led to a constantly shifting dynamic and noticeable letup in progress whenever knowledge had to be extracted from the head of an outgoing developer and incorporated into the onboarding of a new developer. However, this form of rotation has enabled a great breadth of expertise to be invested in the product, which is to be celebrated. We have just completed the move to become a fully co-located team working in eLife’s Cambridge offices, and I for one will be interested to see if this has any impact on our overall productivity.

eLife has not been developing Libero Reviewer in isolation. In September 2017, we announced a partnership with the Collaborative Knowledge Foundation (Coko) and, since then, have been working alongside a number of organisations all using Coko’s PubSweet framework to build their own applications, while simultaneously feeding back any new components for others to reuse. The theory behind the collaboration was to accelerate progress towards a common goal; namely, to transform the way in which science is evaluated, shared and consumed, through the sharing and reuse of components, interfaces and data models. In particular, since the sharing of client-side React components is done regularly and well, the PubSweet framework offers a strong, growing foundation out of the box for any new developer to get started with. Representatives from all of the participating organisations keep in daily contact across several open Mattermost channels and we hold tri-annual meetups where knowledge and best practices can be shared, discussed and agreed upon face-to-face, with better decisions being made for having more voices in the room.

More voices, however, can also mean a decision-making process that depends heavily on compromise. While each organisation shares a mission statement, the applications we are building ultimately sustain very different workflows and users, and fulfil different high-level visions of each application’s purpose. We have seen that, as the applications in the PubSweet ecosystem grow, it is becoming increasingly difficult to rely on a single, unified framework, particularly with regards to server-side support. Notably, on more than one occasion eLife developers have encountered setbacks in the development of the application as a direct result of PubSweet’s shifting base model. Moving forward, it will be important to establish a stable architecture, one which will scale with Libero Reviewer’s expanding codebase, while maintaining our important ties with the PubSweet consortium.

One of the disadvantages of launching a new submission and peer-review system in small increments is that we have had to undertake the added risk of integrating Libero Reviewer with the existing proprietary software that continues to be used to support our editorial workflows. This software represents a ‘black box’ for us, since we have no access to, or knowledge of, its code. Instead, we have been relying upon external developers to write the import script that will a) accept the package we export from Libero Reviewer and b) use the contents to populate a submission within their own system. We were disappointed to learn that our initial requirements could not be technically supported within the parameters of their software, so after a considerable exchange, we arrived at a compromise which has unfortunately resulted in added responsibilities on our own Editorial staff. Most disappointingly, the integration process and subsequent testing significantly delayed our release.

Finally, given the added complexities of the Libero Reviewer project, I have learnt an immense amount in the short time I have been in post. As we progress to the next stages of design and development to support the staff quality control steps and end-to-end workflow for eLife Feature Articles, I take with me a number of lessons that will shape how we work:

• Crucially, I am keen to encourage stakeholders, and particularly the Editorial team, to feedback on designs earlier in the process. They will be power users of the system and can act as a direct proxy for both the authors and editors they interact with on a daily basis. Considering their requirements earlier will help to minimise changing scope later, since we are developing according to Lean principles in thin slices of incremental value.
• We have already established a stakeholder group formed of representatives from each of eLife’s internal departments who have volunteered their time to be part of design workshops and user testing sessions. We hope that this important step will help them feel that their opinions are valued, as well as ensuring we are all aligned behind the same product vision.
• During our final testing of the system prior to launch, we received some last-minute feedback from internal stakeholders which affected the launch window. In future, when performing early-stage prototype testing, we need to find ways of engaging our internal stakeholders to feedback as if we were launching the next day, and better convey the importance of early prototype testing and the feedback we need to gather.
• We should take every opportunity to streamline our current workflows that have been in place since the journal’s inception in 2012. We have the perfect opportunity to deconstruct every aspect of our processes and reevaluate the ways in which we currently operate. Importantly, we should be mindful of inertia and avoid taking things for granted when reassessing why we do things the way we do.
• We should be diligent in gathering user stories and requirements, persistently asking ‘why’ until we understand the underlying problem space, rather than the solution that the current system presents.
• We should ensure that any further points of integration with the legacy software are appropriately de-risked as soon as they are identified, with no technical solutions taken for granted. We should state requirements as extensively and plainly as possible, and clarify up front where necessary.

We are giving ourselves some time post-release to take stock of the codebase, perform any refactoring tasks, diminish technical debt that we acquired in the course of following Lean practices and generally pave the way for the next development cycle. It is important that the foundations upon which we continue to build are as stable as possible, and we anticipate that this will first mean fundamental changes to the application’s architecture.

In terms of design, we are forging ahead with supporting staff quality checks, which will involve the ability to view a read-only version of the information entered into the submission wizard, add comments, request changes and mark all as resolved. In exploring the problem space, we have discovered a great deal of overlap between this step in the editorial workflow and that later during production, when similar requirements are needed for staff and authors to collaborate on proofs ahead of publication.

Beyond this, the internal stakeholder group has determined that developing to facilitate the entire life cycle of a Feature Article is a viable option for the next releasable milestone, requiring the design and technical implementation of a number of core workflows including role assignment and peer review. Armed with all we have learnt, we are incredibly excited to start work on this phase, which will kick off with detailed user-mapping exercises in the coming weeks.

#

We welcome comments, questions and feedback. Please annotate publicly on the article or contact us at innovation [at] elifesciences [dot] org.

Are you interested in contributing to open-source projects to drive forward open science? Register your interest for the eLife Innovation Sprint in September 2019.

Do you have an idea or innovation to share? Send a short outline for a Labs blogpost to innovation [at] elifesciences [dot] org.

For the latest in innovation, eLife Labs and new open-source tools, sign up for our technology and innovation newsletter. You can also follow @eLifeInnovation on Twitter.