Epidemiological characteristics and prevalence rates of research reproducibility across disciplines: a scoping review of articles published in 2018-2019

  1. Kelly D Cobey  Is a corresponding author
  2. Christophe A Fehlmann
  3. Marina Christ Franco
  4. Ana Patricia Ayala
  5. Lindsey Sikora
  6. Danielle B Rice
  7. Chenchen Xu
  8. John PA Ioannidis
  9. Manoj M Lalu
  10. Alixe Ménard
  11. Andrew Neitzel
  12. Bea Nguyen
  13. Nino Tsertsvadze
  14. David Moher
  1. University of Ottawa, Canada
  2. Federal University of Pelotas, Brazil
  3. University of Toronto, Canada
  4. McGill University, Canada
  5. Stanford University, United States
  6. Ottawa Hospital Research Institute, Canada

Abstract

Introduction: Reproducibility is a central tenant of research. We aimed to synthesize the literature on reproducibility and describe its epidemiological characteristics, including how reproducibility is defined and assessed. We also aimed to determine and compare estimates for reproducibility across different fields.

Methods: We conducted a scoping review to identify English language replication studies published between 2018-2019 in economics, education, psychology, health sciences and biomedicine. We searched Medline, Embase, PsycINFO, Cumulative Index of Nursing and Allied Health Literature – CINAHL, Education Source via EBSCOHost, ERIC, EconPapers, International Bibliography of the Social Sciences (IBSS), and EconLit. Documents retrieved were screened in duplicate against our inclusion criteria. We extracted year of publication, number of authors, country of affiliation of the corresponding author, and whether the study was funded. For the individual replication studies, we recorded whether a registered protocol for the replication study was used, whether there was contact between the reproducing team and the original authors, what study design was used, and what the primary outcome was. Finally, we recorded how reproducibilty was defined by the authors, and whether the assessed study(ies) successfully reproduced based on this definition. Extraction was done by a single reviewer and quality controlled by a second reviewer.

Results: Our search identified 11,224 unique documents, of which 47 were included in this review. Most studies were related to either psychology (48.6%) or health sciences (23.7%). Among these 47 documents, 36 described a single reproducibility study while the remaining 11 reported at least two reproducibility studies in the same paper. Less than the half of the studies referred to a registered protocol. There was variability in the definitions of reproduciblity success. In total, across the 47 documents 177 studies were reported. Based on the definition used by the author of each study, 95 of 177 (53.7%) studies reproduced.

Conclusion: This study gives an overview of research across five disciplines that explicitly set out to reproduce previous research. Such reproducibility studies are extremely scarce, the definition of a successfully reproduced study is ambiguous, and the reproducibility rate is overall modest.

Funding: No external funding was received for this work.

Data availability

Data and materials are available on the Open Science Framework (https://osf.io/wn7gm/).

The following data sets were generated

Article and author information

Author details

  1. Kelly D Cobey

    Heart Institute, University of Ottawa, Ottawa, Canada
    For correspondence
    kcobey@ottawaheart.ca
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0003-2797-1686
  2. Christophe A Fehlmann

    School of Epidemiology and Public Health, University of Ottawa, Ottawa, Canada
    Competing interests
    The authors declare that no competing interests exist.
  3. Marina Christ Franco

    School of Dentistry, Federal University of Pelotas, Pelotas, Brazil
    Competing interests
    The authors declare that no competing interests exist.
  4. Ana Patricia Ayala

    Gerstein Science Information Centre, University of Toronto, Toronto, Canada
    Competing interests
    The authors declare that no competing interests exist.
  5. Lindsey Sikora

    Health Sciences Library, University of Ottawa, Ottawa, Canada
    Competing interests
    The authors declare that no competing interests exist.
  6. Danielle B Rice

    Department of Psychology, McGill University, Montreal, Canada
    Competing interests
    The authors declare that no competing interests exist.
  7. Chenchen Xu

    Department of Medicine, University of Ottawa, Ottawa, Canada
    Competing interests
    The authors declare that no competing interests exist.
  8. John PA Ioannidis

    Meta-Research Innovation Center at Stanford, Stanford University, Stanford, United States
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0003-3118-6859
  9. Manoj M Lalu

    Department of Anesthesiology and Pain Medicine, University of Ottawa, Ottawa, Canada
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-0322-382X
  10. Alixe Ménard

    Centre for Journalology, University of Ottawa, Ottawa, Canada
    Competing interests
    The authors declare that no competing interests exist.
  11. Andrew Neitzel

    Department of Medicine, University of Ottawa, Ottawa, Canada
    Competing interests
    The authors declare that no competing interests exist.
  12. Bea Nguyen

    Department of Medicine, University of Ottawa, Ottawa, Canada
    Competing interests
    The authors declare that no competing interests exist.
  13. Nino Tsertsvadze

    Centre for Journalology, Ottawa Hospital Research Institute, Ottawa, Canada
    Competing interests
    The authors declare that no competing interests exist.
  14. David Moher

    Centre for Journalology, Ottawa Hospital Research Institute, Ottawa, Canada
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0003-2434-4206

Funding

The authors declare that there was no funding for this work.

Reviewing Editor

  1. David B Allison, Indiana University, United States

Version history

  1. Preprint posted: March 9, 2022 (view preprint)
  2. Received: March 10, 2022
  3. Accepted: June 20, 2023
  4. Accepted Manuscript published: June 21, 2023 (version 1)
  5. Version of Record published: July 5, 2023 (version 2)

Copyright

© 2023, Cobey et al.

This article is distributed under the terms of the Creative Commons Attribution License permitting unrestricted use and redistribution provided that the original author and source are credited.

Metrics

  • 953
    views
  • 84
    downloads
  • 4
    citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Open citations (links to open the citations from this article in various online reference manager services)

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)

  1. Kelly D Cobey
  2. Christophe A Fehlmann
  3. Marina Christ Franco
  4. Ana Patricia Ayala
  5. Lindsey Sikora
  6. Danielle B Rice
  7. Chenchen Xu
  8. John PA Ioannidis
  9. Manoj M Lalu
  10. Alixe Ménard
  11. Andrew Neitzel
  12. Bea Nguyen
  13. Nino Tsertsvadze
  14. David Moher
(2023)
Epidemiological characteristics and prevalence rates of research reproducibility across disciplines: a scoping review of articles published in 2018-2019
eLife 12:e78518.
https://doi.org/10.7554/eLife.78518

Share this article

https://doi.org/10.7554/eLife.78518

Further reading

    1. Medicine
    Jinjing Chen, Ruoyu Wang ... Jongsook Kemper
    Research Article

    The nuclear receptor, farnesoid X receptor (FXR/NR1H4), is increasingly recognized as a promising drug target for metabolic diseases, including nonalcoholic steatohepatitis (NASH). Protein-coding genes regulated by FXR are well known, but whether FXR also acts through regulation of long non-coding RNAs (lncRNAs), which vastly outnumber protein-coding genes, remains unknown. Utilizing RNA-seq and global run-on sequencing (GRO-seq) analyses in mouse liver, we found that FXR activation affects the expression of many RNA transcripts from chromatin regions bearing enhancer features. Among these we discovered a previously unannotated liver-enriched enhancer-derived lncRNA (eRNA), termed FXR-induced non-coding RNA (Fincor). We show that Fincor is specifically induced by the hammerhead-type FXR agonists, including GW4064 and tropifexor. CRISPR/Cas9-mediated liver-specific knockdown of Fincor in dietary NASH mice reduced the beneficial effects of tropifexor, an FXR agonist currently in clinical trials for NASH and primary biliary cholangitis (PBC), indicating that amelioration of liver fibrosis and inflammation in NASH treatment by tropifexor is mediated in part by Fincor. Overall, our findings highlight that pharmacological activation of FXR by hammerhead-type agonists induces a novel eRNA, Fincor, contributing to the amelioration of NASH in mice. Fincor may represent a new drug target for addressing metabolic disorders, including NASH.

    1. Cell Biology
    2. Medicine
    Chun Wang, Khushpreet Kaur ... Gabriel Mbalaviele
    Research Article

    Chemotherapy is a widely used treatment for a variety of solid and hematological malignancies. Despite its success in improving the survival rate of cancer patients, chemotherapy causes significant toxicity to multiple organs, including the skeleton, but the underlying mechanisms have yet to be elucidated. Using tumor-free mouse models, which are commonly used to assess direct off-target effects of anti-neoplastic therapies, we found that doxorubicin caused massive bone loss in wild-type mice, a phenotype associated with increased number of osteoclasts, leukopenia, elevated serum levels of danger-associated molecular patterns (DAMPs; e.g. cell-free DNA and ATP) and cytokines (e.g. IL-1β and IL-18). Accordingly, doxorubicin activated the absent in melanoma (AIM2) and NLR family pyrin domain containing 3 (NLRP3) inflammasomes in macrophages and neutrophils, causing inflammatory cell death pyroptosis and NETosis, which correlated with its leukopenic effects. Moreover, the effects of this chemotherapeutic agent on cytokine secretion, cell demise, and bone loss were attenuated to various extent in conditions of AIM2 and/or NLRP3 insufficiency. Thus, we found that inflammasomes are key players in bone loss caused by doxorubicin, a finding that may inspire the development of a tailored adjuvant therapy that preserves the quality of this tissue in patients treated with this class of drugs.