The Reproducibility for Everyone (R4E) initiative runs community-led workshops that provide practical solutions for researchers who want to improve the reproducibility of their work. Executive director and co-founder April Clyburne-Sherin discusses progress in the field so far, and the challenges that remain.
April Clyburne-Sherin
Your work is resolutely "by researchers, for other researchers": why this approach?
Every discipline is different, from the tools and methods it uses to the elements that are most challenging to share and build upon. Having a community of researchers fill the needs that they see in their own field makes the curriculum more relevant and up to date. Our workshops are run by working scientists who share their own knowledge, so every time they bring new resources, new tools and new perspectives. A peer talking about their experiences can also inspire people to decide to try something for the first time: because what they learn is not theoretical advice, it's something that someone has actually benefitted from. And finally, it empowers attendees to see that they could contribute to the workshop, pick up the curriculum and run an event themselves.
R4E started with a small group of researchers – including eLife ambassadors – interested in ways to improve research. What made you decide to focus on reproducibility?
It was a nice place to start addressing how we can do science better, because reproducibility touches upon all different aspects of research, from dissemination to staying organised or documenting your work. The topic was widely discussed at the time, yet many researchers didn't have a handle on how to make their research more reproducible. One thing we have learned (from pre-workshop surveys) is that attendees are already motivated to make changes because almost all of them have experienced the pain of not being able to reproduce results – either results from their colleagues, from a published paper, or even their own. There is still a gap in the core training on some of the basics: data management, data sharing, best practices in each field… these things are still very much taught within labs. Some universities are working to fill this gap, which is great, but in the meantime, we want to reach as many researchers as possible with short, practical and accessible training.
What do you think about the recent increase in number of grassroots organisations dedicated to reproducibility?
This is absolutely what we would want to see. What works for you as a researcher is going to be different depending on your discipline, your career phase, your language, where you're based… We need a diversity of grassroots projects because no group can provide everything for everybody. Most initiatives have their own objectives and communities, but they strive to work side-by-side and researchers move between them depending on their needs. In our workshops, we try to create a space for people to share existing knowledge rather than having to recreate it on their own: there's so much out there which researchers can tap into, from tools to platforms to other initiatives.
One concern I have, however, is around sustainability. Right now, you need to have a deliverable of some kind to get money for reproducibility work. Funding is much more difficult to access if you focus on capacity building for diversity or equity or on maintaining a service. These types of projects rely heavily on volunteers, which means that they won't be as diverse as they could be, and that people will burn out. This is a big challenge.
R4E came to be only two years before COVID-19 emerged. What impact do you think the pandemic has had on advocacy work?
We were always a global group, but we became truly global. We were running online sessions before but also a lot of events at conferences: when scientific meetings went virtual, many people joined who could not have attended in the past because of costs or visas, and our audience became much more diverse. I think, coming out of the pandemic, the biggest opportunity for the reproducibility field is more global collaboration. We can connect with and learn from folks all around the world, see how they have used their resources to improve their research, and solve problems together. And now we also have recent, real-life examples of how using open science tools, from electronic lab notebooks to preprints and repositories, can make international research more straightforward. This will be useful for us.
Beyond being important in and of itself, why do you think diversity is essential to improve scientific reproducibility?
The only way we'll find real solutions that work for every researcher is to tap into existing knowledge. And every academic, no matter where they are based, holds some knowledge about how to improve the methods they are using. If we're not working hard as a community to ensure that people of different backgrounds can participate, we will fail to address the problems that research is currently facing.
How is this drive for inclusion shaping the work you do?
Every time we encourage best practice in reproducibility, we need to make sure that low tech and free options are included. People don't really think about the fact that even things like internet access, are not evenly distributed across communities. Sure, it's nice to have all these licences to automate documentation or to back up data; but it's also important to talk about free repositories, how to organise and name computer files or how to back up a written lab notebook.
Within R4E, we also focus on ensuring that our community is a safe and welcoming place, and we have built the governance and infrastructure that we need to grow as a diverse organisation. One of the ways we try to do this is through our stipend programme, to ensure that people have the support they need to participate. That money allows researchers to go to conferences, to be paid for their labour, or to have childcare or internet access to attend training for example.
It's been nearly a decade since the term "reproducibility crisis" was coined: do you see positive changes taking hold in the community?
Yes, for sure. When I first started talking about reproducibility in 2014, it was a niche thing. Now, I think most researchers are familiar with the term. They may still be a little bit confused about what it means for them exactly, but there's never going to be one definition that works for every discipline. This allows us to jump in and say: "You've heard about this, now let us take some of the mystery out of it and help you find solutions for your work." In fact, we try to avoid using the term "reproducibility crisis" at R4E, because we like to focus on opportunities for researchers to improve their practices.
How would you address common criticisms about the pace of reproducibility advances? For example, that papers are not getting more reproducible despite increased awareness, or that training won't be enough as long as top-down pressures are not conducive to best practices?
I don't really agree… My experience is that, right now, transparency is going in the right direction. It's a lot easier to find scientists trying their best to share their research materials compared to five years ago. I think people forget that research is not a fast-moving community. It's rooted in institutions that are moving at a certain speed, and they aren't going to change in a year. You have to keep working through over 10, 20 years. Overall, any complex issue like reproducibility will require multiple interventions at multiple levels. You can’t solve a human problem with just top-down or bottom-up solutions; it's not 'either/or'. We aren't going to get anywhere if we don’t do the things we can do now. There is a gap in training at the moment, and so we have to fill that gap.
You've talked about researchers being keen to improve how they do research: are there areas where it is more difficult for people to change their behavior?
Honestly, I've never perceived strong resistance from the community. When people talk about resistance, they point to surveys showing that scientists aren't sharing data, that they don't use reporting guidelines properly, that they still publish closed access… things like that. But I don't think it's necessarily a sign of resistance. Many factors go into how and where materials are shared, and it's never just one person making the decision. Usually, it's a group of people having to land on "this is what we had time and money for, this is what everyone could get on board with". The way academics might think about how to stay organised, how to document, how to automate and how to disseminate their work will all be influenced by what they have access to, what their mandates are, and what they are rewarded for. We just need to keep chipping away at it, to make it easier and easier for people to use a range of solutions. We're going in the right direction.
How do you address the emotional impact of reproducibility work?
I'm really glad you asked this, because we tend to forget to think through how people feel about their research and reproducibility. Researchers will attend a training session and become aware of all sorts of problems and all sorts of solutions. And they may feel badly about having participated in some these problems, but also overwhelmed by the changes that are needed.
One thing that often happens in reproducibility workshops, for example, is what I call 'Git Guilt'. Researchers get told that they should learn how to use Git, which is a software that tracks changes in code. They leave the workshop and they can't figure it out, so they feel badly about not adopting this tool they were told they should use if they really cared about reproducibility. Instead of making recommendations that are hard to adopt, it's important to talk about the purpose of a specific tool and to explain how people can make similar improvement to their research through different solutions. So instead of using Git, researchers can track their changes in a rich README file, or with the version control that is built into the software they already have.
No one is perfect. No one is doing all the things that somebody may claim are the solution for reproducibility. We try to focus on encouraging people to make one change at the time, to start with something small that they can control. Rather than becoming overwhelmed, this allows them to feel they are on a journey to improve their research.
April Clyburne-Sherin was interviewed by Elsa Loissel, Associate Features Editor, eLife.
