The first results from the Reproducibility Project: Cancer Biology have been published in eLife. The aim of the project, which is a collaboration between the Center for Open Science and Science Exchange, is to assess reproducibility in cancer biology, and to identify what influences its success or failure in science more generally.
An eLife editorial published to coincide states: “Reproducibility is a cornerstone of science, and the development of new drugs and medical treatments relies on the results of preclinical research being reproducible. In recent years, however, the validity of published findings in a number of areas of scientific research, including cancer research, have been called into question (Begley and Ellis, 2012; Baker, 2016)”.
At the same time, replicating other researchers’ work can be onerous and there is little incentive for researchers to do so. Many journals prefer to publish new results.
“The first five papers are part of a substantial effort to study reproducibility in cancer biology in a transparent way,” says eLife Editor-in-Chief Randy Schekman. “We aim to bolster this effort by applying eLife’s rigorous and consultative editorial process to the evaluation and peer review of the resulting papers.”
“This is an experiment. Every approach to assessing reproducibility has strengths and weaknesses. With over 20 further papers to publish, we do not yet know whether the approach taken in this project is the most effective. However, these initial papers will encourage discussion of these issues and the final analyses will highlight factors that influence reproducibility and the best ways to evaluate it,” he says.
Tim Errington from the Center for Open Science says: “These first five papers show that achieving reproducibility is hard and there is room for improvement. We can already see how variations in the availability of critical information and processes can impact how challenging it is to replicate projects and results.”
The replication studies in the cancer biology project are presented in two distinct phases. First, the project leaders developed protocols for each study and shared those with original authors for informal review, before submitting the protocols as “Registered Reports” to eLife for formal peer review.
Registered Reports are an opportunity to get expert feedback from reviewers to maximize the quality of the experimental design and methodology.
Next, Replication Studies are conducted as specified in the Registered Reports and are also subject to eLife peer review.
“We are proud to facilitate this landmark project. Science Exchange provides efficient access via our unique marketplace to the world-class scientific expertise required to independently conduct these studies,” says founder and CEO of Science Exchange Elizabeth Iorns.
The assessment of eLife editors is that of the first five Replication Studies, two broadly supported key conclusions from the original studies, two were inconclusive due to technical problems with certain key experiments, and one failed to reproduce the findings of the original study.
The editorial concludes: “We will publish more Replication Studies over the months ahead and, at the conclusion of the project, a meta-analysis of all the studies. While we wait for this, it is important not to overinterpret the results. Already it is clear that nuanced interpretations are necessary, not black and white conclusions about which studies reproduced and which did not. It is also clear that this approach to testing reproducibility remains an experiment, with advantages and disadvantages, including the fact that it sometimes yields results that cannot be interpreted.”
The original studies were published between 2010 and 2012. By the end of the project, around 30 Replication Studies will have been reviewed by eLife.
##
Media contacts
Emily Packer
eLife
e.packer@elifesciences.org
+441223855373
About
About the Center for Open Science
The Center for Open Science (COS) is a non-profit technology startup founded in 2013 with a mission to increase openness, integrity, and reproducibility of scientific research. COS pursues this mission by building communities around open science practices, supporting metascience research, and developing and maintaining free, open source software tools. The Open Science Framework (OSF), COS’s flagship product, is a web application that connects and supports the research workflow, enabling scientists to increase the efficiency and effectiveness of their research. Researchers use the OSF to collaborate, document, archive, share, and register research projects, materials, and data. Learn more at cos.io and osf.io.
About Science Exchange
Science Exchange is the world’s leading marketplace for scientific research services. The company provides secure access to a network of over 3000 screened and verified contract research organizations (CROs), academic labs, and government facilities that are available to conduct experiments on the behalf of scientists. The Science Exchange platform has been used by scientists from over 2,500 different companies and organizations, solving one of the most significant challenges facing the highly-trained researchers at these companies: time and resources spent identifying and managing outsourced research projects. To date, Science Exchange has raised over $30 million from Maverick Capital Ventures, Union Square Ventures, Index Ventures, OATV, the YC Continuity Fund, and others. For more information visit www.scienceexchange.com.
About eLife
eLife is a unique collaboration between the funders and practitioners of research to improve the way important research is selected, presented, and shared. eLife publishes outstanding works across the life sciences and biomedicine — from basic biological research to applied, translational, and clinical studies. All papers are selected by active scientists in the research community. Decisions and responses are agreed by the reviewers and consolidated by the Reviewing Editor into a single, clear set of instructions for authors, removing the need for laborious cycles of revision and allowing authors to publish their findings quickly. eLife is supported by the Howard Hughes Medical Institute, the Max Planck Society and the Wellcome Trust. Learn more at elifesciences.org.
